TA-Deployment-Broker/docs/deployment.md

Deployment

This service runs active-active on both webserver VMs. Both installations use the same MariaDB Galera database, Gitea application, package registry, and secrets. The containers use Linux host networking so 127.0.0.1:3306 reaches each webserver's local HAProxy listener.

1. DNS and TLS

Create tapm.scity.us in PowerDNS and point it at the existing HAProxy frontend. Install the certificate using the existing certificate automation. Set the broker to listen on all addresses on both webservers:

TAPM_LISTEN_ADDR=0.0.0.0:8080

The wildcard bind accepts connections through each node's physical address and through the Keepalived address 10.10.1.120 on whichever node currently owns it. The container health check independently uses 127.0.0.1:8080. Restrict port 8080 to the load-balancer and trusted management addresses. If Nginx terminates TLS locally and neither direct nor Keepalived access is required, the broker can instead bind to 127.0.0.1:8080.

An example virtual host is in deploy/nginx/tapm.scity.us.conf. Adjust only the certificate paths if the local convention differs.

2. MariaDB Galera

Create a dedicated database and application users limited to the two webserver addresses. Run this once against the cluster as a database administrator, replacing the password with the same locally generated value in both accounts:

CREATE DATABASE tapm_broker
    CHARACTER SET utf8mb4
    COLLATE utf8mb4_unicode_ci;
CREATE USER 'tapm'@'10.10.1.121' IDENTIFIED BY 'replace-with-a-random-password';
CREATE USER 'tapm'@'10.10.1.122' IDENTIFIED BY 'replace-with-a-random-password';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
    ON tapm_broker.* TO 'tapm'@'10.10.1.121';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
    ON tapm_broker.* TO 'tapm'@'10.10.1.122';
FLUSH PRIVILEGES;

The Keepalived address 10.10.1.120 is not included because outbound HAProxy connections normally originate from the physical webserver address. Add a third account for 10.10.1.120 only if a connection test or the HAProxy configuration proves that it explicitly uses the virtual address as its source.

3. Gitea accounts and OAuth

Create these separately:

1. The taiadmin technician account used to sign in to the portal.
2. A tapm-packages service account with read access only to the private TAI/files package owner. Create a personal access token for this account and store it only in the broker environment file.
3. A tapm-publisher service account with write:package access. Its token is used only by authenticated portal uploads.
4. An OAuth2 application with callback URL https://tapm.scity.us/auth/callback.

The OAuth client secret authenticates the portal to Gitea. The read token is used only while proxying an authorized file, and the write token is used only for authenticated uploads. None are returned to a technician or Proxmox host. The broker requests only Gitea's read:user OAuth scope, which it uses to confirm the signed-in username against TAPM_ALLOWED_GITEA_USERS.

4. Broker configuration

On each webserver:

git clone https://git.scity.us/TAI/TA-Deployment-Broker.git
cd TA-Deployment-Broker
cp .env.example .env
chmod 600 .env

Edit .env and use the same secrets on both webservers. The listen address is the one node-specific value. Generate the cookie secret with:

openssl rand -base64 48

The database DSN must retain parseTime=true, multiStatements=true, and the encoded UTC time_zone setting from .env.example. Its address remains 127.0.0.1:3306, which uses the local HAProxy frontend rather than selecting a single Galera member.

5. Build and migrate

Build the image on each webserver:

docker compose build

Apply migrations from only one webserver:

docker compose run --rm migrate

Migrations use a MariaDB advisory lock, so an accidental simultaneous run cannot apply the same migration twice.

6. Start both instances

On both webservers:

docker compose up -d broker
curl --fail "http://127.0.0.1:8080/health/ready"

There is no local application state and no session affinity requirement.

For later updates, run the repository update helper from either webserver:

/opt/idssys/ta-deployment-broker/update.sh

The helper updates and health-checks the local broker first, then connects as TAPM_UPDATE_SSH_USER to each comma-separated host in TAPM_UPDATE_PEERS and performs the same local-only check there. This provides a sequential rolling update from any node. An unreachable peer is skipped with a warning; a reachable peer whose update fails causes the command to fail. The SSH user defaults to root when omitted.

Configure each webserver's .env with its other broker:

# Webserver-Node1
TAPM_UPDATE_PEERS=10.10.1.122
TAPM_UPDATE_SSH_USER=root

# Webserver-Node2
TAPM_UPDATE_PEERS=10.10.1.121
TAPM_UPDATE_SSH_USER=root

Additional peers can be listed with commas. The updater reads only these named settings and does not source .env as executable shell code.

Each node fetches and compares its current tracked branch with its upstream. If both commits match, that node's Docker services are untouched. When an update exists, the helper fast-forwards the branch, rebuilds the image, applies pending migrations, recreates the broker container, and waits up to 150 seconds for its Docker health check. It refuses tracked local changes and detached, untracked, locally-ahead, or diverged branches.

7. HAProxy health check

The load balancer should consider an origin healthy only when GET /health/ready returns HTTP 200. /health/live checks only the process; /health/ready also verifies Galera connectivity.

Both healthy webservers can receive traffic. leastconn is useful because package uploads and downloads stay open longer than portal requests.

8. Publish a protected package

The normal workflow is Protected packages → Update a package in the portal. The package ID is the stable name used by the broker, ProxMenu, and Gitea; a separate registry package name is not required. The broker streams the replacement directly to Gitea, calculates SHA-256, updates the catalog, removes the prior Gitea version, and writes audit events without buffering the installer on local disk.

An authorization grants access to the stable package ID, not a particular version. Existing active codes and download sessions resolve the package's current registry version and checksum, so they automatically use a replacement after it is registered.

The load balancer or Nginx must allow request bodies up to TAPM_MAX_UPLOAD_BYTES and use a sufficiently long request timeout. The default is 1 GiB and 30 minutes.

For recovery, a package can still be published directly:

curl --user 'publisher:PACKAGE_WRITE_TOKEN' \
  --upload-file ./SentinelAgent_linux_x86_64_v26_1_1_31.deb \
  'https://git.scity.us/api/packages/TAI/generic/sentinelone-linux/26.1.1.31/SentinelAgent_linux_x86_64_v26_1_1_31.deb'

Calculate its checksum:

sha256sum SentinelAgent_linux_x86_64_v26_1_1_31.deb

Sign in at https://tapm.scity.us, expand Register package metadata manually, enter the matching values, and enable it. The broker never uses the technician's Gitea credentials for package operations.

9. Backup and rotation

• Include tapm_broker in the existing Galera backup policy.
• Audit records are retained indefinitely unless an administrator establishes a database retention policy. The Codes view previews the newest 15 records; the Audit view can filter retained history by event, customer/deployment label, user, host, package, source IP, or details and displays up to the newest 250 matching events.
• Keep .env outside Git and readable only by the service administrator.
• Rotate both Gitea package tokens and the OAuth secret if either webserver is compromised.
• Changing TAPM_COOKIE_SECRET signs out portal users; it does not invalidate active deployment codes.
• Revoke active deployment authorizations from the portal when a deployment ends early.