TA-Deployment-Broker/docs/deployment.md

Deployment

This service is intended to run on both webserver VMs, with only the active webserver's broker container started. Both installations use the same MariaDB Galera database, Gitea application, package registry, and secrets. The containers use Linux host networking so 127.0.0.1:3306 reaches each webserver's local HAProxy listener.

1. DNS and TLS

Create tapm.scity.us in PowerDNS and point it at the existing HAProxy frontend. Install the certificate using the existing certificate automation. The broker only listens on 127.0.0.1:8080; Nginx terminates TLS.

An example virtual host is in deploy/nginx/tapm.scity.us.conf. Adjust only the certificate paths if the local convention differs.

2. MariaDB Galera

Create a dedicated database and application users limited to the two webserver addresses. Run this once against the cluster as a database administrator, replacing the password with the same locally generated value in both accounts:

CREATE DATABASE tapm_broker
    CHARACTER SET utf8mb4
    COLLATE utf8mb4_unicode_ci;
CREATE USER 'tapm'@'10.10.1.121' IDENTIFIED BY 'replace-with-a-random-password';
CREATE USER 'tapm'@'10.10.1.122' IDENTIFIED BY 'replace-with-a-random-password';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
    ON tapm_broker.* TO 'tapm'@'10.10.1.121';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
    ON tapm_broker.* TO 'tapm'@'10.10.1.122';
FLUSH PRIVILEGES;

The Keepalived address 10.10.1.120 is not included because outbound HAProxy connections normally originate from the physical webserver address. Add a third account for 10.10.1.120 only if a connection test or the HAProxy configuration proves that it explicitly uses the virtual address as its source.

3. Gitea accounts and OAuth

Create these separately:

1. The taiadmin technician account used to sign in to the portal.
2. A tapm-packages service account with read access only to the private TAI/files package owner. Create a personal access token for this account and store it only in the broker environment file.
3. An OAuth2 application with callback URL https://tapm.scity.us/auth/callback.

The OAuth client secret authenticates the portal to Gitea. The package token is used only server-side while proxying an authorized file. Neither value is returned to a technician or Proxmox host.

4. Broker configuration

On each webserver:

git clone https://git.scity.us/TAI/TA-Deployment-Broker.git
cd TA-Deployment-Broker
cp .env.example .env
chmod 600 .env

Edit .env and use the same values on both webservers. Generate the cookie secret with:

openssl rand -base64 48

The database DSN must retain parseTime=true, multiStatements=true, and the encoded UTC time_zone setting from .env.example. Its address remains 127.0.0.1:3306, which uses the local HAProxy frontend rather than selecting a single Galera member.

5. Build and migrate

Build the image on each webserver:

docker compose build

Apply migrations from only one webserver:

docker compose run --rm migrate

Migrations use a MariaDB advisory lock, so an accidental simultaneous run cannot apply the same migration twice.

6. Start the active instance

On the active webserver:

docker compose up -d broker
curl --fail http://127.0.0.1:8080/health/ready

On the standby webserver, keep the image and .env current but leave the broker stopped. The existing failover automation can run:

docker compose stop broker

on the old active VM, followed by:

docker compose up -d broker

on the new active VM. There is no local application state to transfer.

7. HAProxy health check

The load balancer should consider an origin healthy only when GET /health/ready returns HTTP 200. /health/live checks only the process; /health/ready also verifies Galera connectivity.

Only the active webserver should be eligible for traffic. Retaining a single active origin avoids duplicate service management while Galera preserves sessions and authorization state across failover.

8. Publish a protected package

Publish files to the private Gitea Generic Package Registry:

curl --user 'publisher:PACKAGE_WRITE_TOKEN' \
  --upload-file ./SentinelAgent_linux_x86_64.deb \
  'https://git.scity.us/api/packages/TAI/generic/sentinelone-linux/26.1.1.31/SentinelAgent_linux_x86_64.deb'

Calculate its checksum:

sha256sum SentinelAgent_linux_x86_64.deb

Sign in at https://tapm.scity.us, enter the matching registry package name, version, filename, and SHA-256, then enable it. The broker intentionally does not browse the registry with the technician's credentials.

9. Backup and rotation

• Include tapm_broker in the existing Galera backup policy.
• Keep .env outside Git and readable only by the service administrator.
• Rotate the Gitea package token and OAuth secret if either webserver is compromised.
• Changing TAPM_COOKIE_SECRET signs out portal users; it does not invalidate active deployment codes.
• Revoke active deployment authorizations from the portal when a deployment ends early.