TA-Deployment-Broker/docs/deployment.md

# Deployment

This service runs active-active on both webserver VMs. Both installations use
the same MariaDB Galera database, Gitea application, package registry, and
secrets. The containers use Linux host networking so `127.0.0.1:3306` reaches
each webserver's local HAProxy listener.

## 1. DNS and TLS

Create `tapm.scity.us` in PowerDNS and point it at the existing HAProxy
frontend. Install the certificate using the existing certificate automation.
Set `TAPM_LISTEN_ADDR` to each node's physical address when the load balancer
connects directly to port 8080:

- Webserver-Node1: `10.10.1.121:8080`
- Webserver-Node2: `10.10.1.122:8080`

Restrict port 8080 to the load-balancer addresses. If Nginx terminates TLS on
each webserver instead, keep the broker on `127.0.0.1:8080`.

An example virtual host is in
[`deploy/nginx/tapm.scity.us.conf`](../deploy/nginx/tapm.scity.us.conf).
Adjust only the certificate paths if the local convention differs.

## 2. MariaDB Galera

Create a dedicated database and application users limited to the two webserver
addresses. Run this once against the cluster as a database administrator,
replacing the password with the same locally generated value in both accounts:

```sql
CREATE DATABASE tapm_broker
    CHARACTER SET utf8mb4
    COLLATE utf8mb4_unicode_ci;
CREATE USER 'tapm'@'10.10.1.121' IDENTIFIED BY 'replace-with-a-random-password';
CREATE USER 'tapm'@'10.10.1.122' IDENTIFIED BY 'replace-with-a-random-password';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
    ON tapm_broker.* TO 'tapm'@'10.10.1.121';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
    ON tapm_broker.* TO 'tapm'@'10.10.1.122';
FLUSH PRIVILEGES;
```

The Keepalived address `10.10.1.120` is not included because outbound HAProxy
connections normally originate from the physical webserver address. Add a
third account for `10.10.1.120` only if a connection test or the HAProxy
configuration proves that it explicitly uses the virtual address as its source.

## 3. Gitea accounts and OAuth

Create these separately:

1. The `taiadmin` technician account used to sign in to the portal.
2. A `tapm-packages` service account with read access only to the private
   `TAI/files` package owner. Create a personal access token for this account
   and store it only in the broker environment file.
3. A `tapm-publisher` service account with `write:package` access. Its token is
   used only by authenticated portal uploads.
4. An OAuth2 application with callback URL
   `https://tapm.scity.us/auth/callback`.

The OAuth client secret authenticates the portal to Gitea. The read token is
used only while proxying an authorized file, and the write token is used only
for authenticated uploads. None are returned to a technician or Proxmox host.
The broker requests only Gitea's `read:user` OAuth scope, which it uses to
confirm the signed-in username against `TAPM_ALLOWED_GITEA_USERS`.

## 4. Broker configuration

On each webserver:

```sh
git clone https://git.scity.us/TAI/TA-Deployment-Broker.git
cd TA-Deployment-Broker
cp .env.example .env
chmod 600 .env
```

Edit `.env` and use the same secrets on both webservers. The listen address is
the one node-specific value. Generate the cookie secret with:

```sh
openssl rand -base64 48
```

The database DSN must retain `parseTime=true`, `multiStatements=true`, and the
encoded UTC `time_zone` setting from `.env.example`. Its address remains
`127.0.0.1:3306`, which uses the local HAProxy frontend rather than selecting a
single Galera member.

## 5. Build and migrate

Build the image on each webserver:

```sh
docker compose build
```

Apply migrations from only one webserver:

```sh
docker compose run --rm migrate
```

Migrations use a MariaDB advisory lock, so an accidental simultaneous run
cannot apply the same migration twice.

## 6. Start both instances

On both webservers:

```sh
docker compose up -d broker
curl --fail "http://${TAPM_LISTEN_ADDR}/health/ready"
```

There is no local application state and no session affinity requirement.

## 7. HAProxy health check

The load balancer should consider an origin healthy only when
`GET /health/ready` returns HTTP 200. `/health/live` checks only the process;
`/health/ready` also verifies Galera connectivity.

Both healthy webservers can receive traffic. `leastconn` is useful because
package uploads and downloads stay open longer than portal requests.

## 8. Publish a protected package

The normal workflow is **Protected packages → Update a package** in the portal.
The package ID is the stable name used by the broker, ProxMenu, and Gitea; a
separate registry package name is not required. The broker streams the
replacement directly to Gitea, calculates SHA-256, updates the catalog, removes
the prior Gitea version, and writes audit events without buffering the installer
on local disk.

An authorization grants access to the stable package ID, not a particular
version. Existing active codes and download sessions resolve the package's
current registry version and checksum, so they automatically use a replacement
after it is registered.

The load balancer or Nginx must allow request bodies up to
`TAPM_MAX_UPLOAD_BYTES` and use a sufficiently long request timeout. The
default is 1 GiB and 30 minutes.

For recovery, a package can still be published directly:

```sh
curl --user 'publisher:PACKAGE_WRITE_TOKEN' \
  --upload-file ./SentinelAgent_linux_x86_64_v26_1_1_31.deb \
  'https://git.scity.us/api/packages/TAI/generic/sentinelone-linux/26.1.1.31/SentinelAgent_linux_x86_64_v26_1_1_31.deb'
```

Calculate its checksum:

```sh
sha256sum SentinelAgent_linux_x86_64_v26_1_1_31.deb
```

Sign in at `https://tapm.scity.us`, expand **Register package metadata
manually**, enter the matching values, and enable it. The broker never uses the
technician's Gitea credentials for package operations.

## 9. Backup and rotation

- Include `tapm_broker` in the existing Galera backup policy.
- Audit records are retained indefinitely unless an administrator establishes a
  database retention policy. The Codes view previews the newest 15 records; the
  Audit view can filter the retained history and displays up to the newest 250
  matching events.
- Keep `.env` outside Git and readable only by the service administrator.
- Rotate both Gitea package tokens and the OAuth secret if either webserver is
  compromised.
- Changing `TAPM_COOKIE_SECRET` signs out portal users; it does not invalidate
  active deployment codes.
- Revoke active deployment authorizations from the portal when a deployment
  ends early.