TA-Deployment-Broker/docs/deployment.md

Deployment

This service runs active-active on both webserver VMs. Both installations use the same MariaDB Galera database, Gitea application, package registry, and secrets. The containers use Linux host networking so 127.0.0.1:3306 reaches each webserver's local HAProxy listener.

1. DNS and TLS

Create tapm.scity.us in PowerDNS and point it at the existing HAProxy frontend. Install the certificate using the existing certificate automation. Set TAPM_LISTEN_ADDR to each node's physical address when the load balancer connects directly to port 8080:

• Webserver-Node1: 10.10.1.121:8080
• Webserver-Node2: 10.10.1.122:8080

Restrict port 8080 to the load-balancer addresses. If Nginx terminates TLS on each webserver instead, keep the broker on 127.0.0.1:8080.

An example virtual host is in deploy/nginx/tapm.scity.us.conf. Adjust only the certificate paths if the local convention differs.

2. MariaDB Galera

Create a dedicated database and application users limited to the two webserver addresses. Run this once against the cluster as a database administrator, replacing the password with the same locally generated value in both accounts:

CREATE DATABASE tapm_broker
    CHARACTER SET utf8mb4
    COLLATE utf8mb4_unicode_ci;
CREATE USER 'tapm'@'10.10.1.121' IDENTIFIED BY 'replace-with-a-random-password';
CREATE USER 'tapm'@'10.10.1.122' IDENTIFIED BY 'replace-with-a-random-password';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
    ON tapm_broker.* TO 'tapm'@'10.10.1.121';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
    ON tapm_broker.* TO 'tapm'@'10.10.1.122';
FLUSH PRIVILEGES;

The Keepalived address 10.10.1.120 is not included because outbound HAProxy connections normally originate from the physical webserver address. Add a third account for 10.10.1.120 only if a connection test or the HAProxy configuration proves that it explicitly uses the virtual address as its source.

3. Gitea accounts and OAuth

Create these separately:

1. The taiadmin technician account used to sign in to the portal.
2. A tapm-packages service account with read access only to the private TAI/files package owner. Create a personal access token for this account and store it only in the broker environment file.
3. A tapm-publisher service account with write:package access. Its token is used only by authenticated portal uploads.
4. An OAuth2 application with callback URL https://tapm.scity.us/auth/callback.

The OAuth client secret authenticates the portal to Gitea. The read token is used only while proxying an authorized file, and the write token is used only for authenticated uploads. None are returned to a technician or Proxmox host. The broker requests only Gitea's read:user OAuth scope, which it uses to confirm the signed-in username against TAPM_ALLOWED_GITEA_USERS.

4. Broker configuration

On each webserver:

git clone https://git.scity.us/TAI/TA-Deployment-Broker.git
cd TA-Deployment-Broker
cp .env.example .env
chmod 600 .env

Edit .env and use the same secrets on both webservers. The listen address is the one node-specific value. Generate the cookie secret with:

openssl rand -base64 48

The database DSN must retain parseTime=true, multiStatements=true, and the encoded UTC time_zone setting from .env.example. Its address remains 127.0.0.1:3306, which uses the local HAProxy frontend rather than selecting a single Galera member.

5. Build and migrate

Build the image on each webserver:

docker compose build

Apply migrations from only one webserver:

docker compose run --rm migrate

Migrations use a MariaDB advisory lock, so an accidental simultaneous run cannot apply the same migration twice.

6. Start both instances

On both webservers:

docker compose up -d broker
curl --fail "http://${TAPM_LISTEN_ADDR}/health/ready"

There is no local application state and no session affinity requirement.

7. HAProxy health check

The load balancer should consider an origin healthy only when GET /health/ready returns HTTP 200. /health/live checks only the process; /health/ready also verifies Galera connectivity.

Both healthy webservers can receive traffic. leastconn is useful because package uploads and downloads stay open longer than portal requests.

8. Publish a protected package

The normal workflow is Protected packages → Update a package in the portal. The package ID is the stable name used by the broker, ProxMenu, and Gitea; a separate registry package name is not required. The broker streams the replacement directly to Gitea, calculates SHA-256, updates the catalog, removes the prior Gitea version, and writes audit events without buffering the installer on local disk.

An authorization grants access to the stable package ID, not a particular version. Existing active codes and download sessions resolve the package's current registry version and checksum, so they automatically use a replacement after it is registered.

The load balancer or Nginx must allow request bodies up to TAPM_MAX_UPLOAD_BYTES and use a sufficiently long request timeout. The default is 1 GiB and 30 minutes.

For recovery, a package can still be published directly:

curl --user 'publisher:PACKAGE_WRITE_TOKEN' \
  --upload-file ./SentinelAgent_linux_x86_64_v26_1_1_31.deb \
  'https://git.scity.us/api/packages/TAI/generic/sentinelone-linux/26.1.1.31/SentinelAgent_linux_x86_64_v26_1_1_31.deb'

Calculate its checksum:

sha256sum SentinelAgent_linux_x86_64_v26_1_1_31.deb

Sign in at https://tapm.scity.us, expand Register package metadata manually, enter the matching values, and enable it. The broker never uses the technician's Gitea credentials for package operations.

9. Backup and rotation

• Include tapm_broker in the existing Galera backup policy.
• Audit records are retained indefinitely unless an administrator establishes a database retention policy. The Codes view previews the newest 15 records; the Audit view can filter the retained history and displays up to the newest 250 matching events.
• Keep .env outside Git and readable only by the service administrator.
• Rotate both Gitea package tokens and the OAuth secret if either webserver is compromised.
• Changing TAPM_COOKIE_SECRET signs out portal users; it does not invalidate active deployment codes.
• Revoke active deployment authorizations from the portal when a deployment ends early.