TA-Deployment-Broker/docs/deployment.md

# Deployment

This service is intended to run on both webserver VMs, with only the active
webserver's broker container started. Both installations use the same MariaDB
Galera database, Gitea application, package registry, and secrets. The
containers use Linux host networking so `127.0.0.1:3306` reaches each
webserver's local HAProxy listener.

## 1. DNS and TLS

Create `tapm.scity.us` in PowerDNS and point it at the existing HAProxy
frontend. Install the certificate using the existing certificate automation.
The broker only listens on `127.0.0.1:8080`; Nginx terminates TLS.

An example virtual host is in
[`deploy/nginx/tapm.scity.us.conf`](../deploy/nginx/tapm.scity.us.conf).
Adjust only the certificate paths if the local convention differs.

## 2. MariaDB Galera

Create a dedicated database and application users limited to the two webserver
addresses. Run this once against the cluster as a database administrator,
replacing the password with the same locally generated value in both accounts:

```sql
CREATE DATABASE tapm_broker
    CHARACTER SET utf8mb4
    COLLATE utf8mb4_unicode_ci;
CREATE USER 'tapm'@'10.10.1.121' IDENTIFIED BY 'replace-with-a-random-password';
CREATE USER 'tapm'@'10.10.1.122' IDENTIFIED BY 'replace-with-a-random-password';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
    ON tapm_broker.* TO 'tapm'@'10.10.1.121';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
    ON tapm_broker.* TO 'tapm'@'10.10.1.122';
FLUSH PRIVILEGES;
```

The Keepalived address `10.10.1.120` is not included because outbound HAProxy
connections normally originate from the physical webserver address. Add a
third account for `10.10.1.120` only if a connection test or the HAProxy
configuration proves that it explicitly uses the virtual address as its source.

## 3. Gitea accounts and OAuth

Create these separately:

1. The `taiadmin` technician account used to sign in to the portal.
2. A `tapm-packages` service account with read access only to the private
   `TAI/files` package owner. Create a personal access token for this account
   and store it only in the broker environment file.
3. An OAuth2 application with callback URL
   `https://tapm.scity.us/auth/callback`.

The OAuth client secret authenticates the portal to Gitea. The package token
is used only server-side while proxying an authorized file. Neither value is
returned to a technician or Proxmox host. The broker requests only Gitea's
`read:user` OAuth scope, which it uses to confirm the signed-in username
against `TAPM_ALLOWED_GITEA_USERS`.

## 4. Broker configuration

On each webserver:

```sh
git clone https://git.scity.us/TAI/TA-Deployment-Broker.git
cd TA-Deployment-Broker
cp .env.example .env
chmod 600 .env
```

Edit `.env` and use the same values on both webservers. Generate the cookie
secret with:

```sh
openssl rand -base64 48
```

The database DSN must retain `parseTime=true`, `multiStatements=true`, and the
encoded UTC `time_zone` setting from `.env.example`. Its address remains
`127.0.0.1:3306`, which uses the local HAProxy frontend rather than selecting a
single Galera member.

## 5. Build and migrate

Build the image on each webserver:

```sh
docker compose build
```

Apply migrations from only one webserver:

```sh
docker compose run --rm migrate
```

Migrations use a MariaDB advisory lock, so an accidental simultaneous run
cannot apply the same migration twice.

## 6. Start the active instance

On the active webserver:

```sh
docker compose up -d broker
curl --fail http://127.0.0.1:8080/health/ready
```

On the standby webserver, keep the image and `.env` current but leave the
broker stopped. The existing failover automation can run:

```sh
docker compose stop broker
```

on the old active VM, followed by:

```sh
docker compose up -d broker
```

on the new active VM. There is no local application state to transfer.

## 7. HAProxy health check

The load balancer should consider an origin healthy only when
`GET /health/ready` returns HTTP 200. `/health/live` checks only the process;
`/health/ready` also verifies Galera connectivity.

Only the active webserver should be eligible for traffic. Retaining a single
active origin avoids duplicate service management while Galera preserves
sessions and authorization state across failover.

## 8. Publish a protected package

Publish files to the private Gitea Generic Package Registry:

```sh
curl --user 'publisher:PACKAGE_WRITE_TOKEN' \
  --upload-file ./SentinelAgent_linux_x86_64.deb \
  'https://git.scity.us/api/packages/TAI/generic/sentinelone-linux/26.1.1.31/SentinelAgent_linux_x86_64.deb'
```

Calculate its checksum:

```sh
sha256sum SentinelAgent_linux_x86_64.deb
```

Sign in at `https://tapm.scity.us`, enter the matching registry package name,
version, filename, and SHA-256, then enable it. The broker intentionally does
not browse the registry with the technician's credentials.

## 9. Backup and rotation

- Include `tapm_broker` in the existing Galera backup policy.
- Keep `.env` outside Git and readable only by the service administrator.
- Rotate the Gitea package token and OAuth secret if either webserver is
  compromised.
- Changing `TAPM_COOKIE_SECRET` signs out portal users; it does not invalidate
  active deployment codes.
- Revoke active deployment authorizations from the portal when a deployment
  ends early.