# Deployment

This service runs active-active on both webserver VMs. Both installations use
the same MariaDB Galera database, Gitea application, package registry, and
secrets. The containers use Linux host networking so `127.0.0.1:3306` reaches
each webserver's local HAProxy listener.

## 1. DNS and TLS

Create `tapm.scity.us` in PowerDNS and point it at the existing HAProxy
frontend. Install the certificate using the existing certificate automation.
Set `TAPM_LISTEN_ADDR` to each node's physical address when the load balancer
connects directly to port 8080:

- Webserver-Node1: `10.10.1.121:8080`
- Webserver-Node2: `10.10.1.122:8080`

Restrict port 8080 to the load-balancer addresses. If Nginx terminates TLS on
each webserver instead, keep the broker on `127.0.0.1:8080`.

An example virtual host is in
[`deploy/nginx/tapm.scity.us.conf`](../deploy/nginx/tapm.scity.us.conf).
Adjust only the certificate paths if the local convention differs.

## 2. MariaDB Galera

Create a dedicated database and application users limited to the two webserver
addresses. Run this once against the cluster as a database administrator,
replacing the password with the same locally generated value in both accounts:

```sql
CREATE DATABASE tapm_broker
    CHARACTER SET utf8mb4
    COLLATE utf8mb4_unicode_ci;
CREATE USER 'tapm'@'10.10.1.121' IDENTIFIED BY 'replace-with-a-random-password';
CREATE USER 'tapm'@'10.10.1.122' IDENTIFIED BY 'replace-with-a-random-password';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
    ON tapm_broker.* TO 'tapm'@'10.10.1.121';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
    ON tapm_broker.* TO 'tapm'@'10.10.1.122';
FLUSH PRIVILEGES;
```

The Keepalived address `10.10.1.120` is not included because outbound HAProxy
connections normally originate from the physical webserver address. Add a
third account for `10.10.1.120` only if a connection test or the HAProxy
configuration proves that it explicitly uses the virtual address as its source.

## 3. Gitea accounts and OAuth

Create these separately:

1. The `taiadmin` technician account used to sign in to the portal.
2. A `tapm-packages` service account with read access only to the private
   `TAI/files` package owner. Create a personal access token for this account
   and store it only in the broker environment file.
3. A `tapm-publisher` service account with `write:package` access. Its token is
   used only by authenticated portal uploads.
4. An OAuth2 application with callback URL
   `https://tapm.scity.us/auth/callback`.

The OAuth client secret authenticates the portal to Gitea. The read token is
used only while proxying an authorized file, and the write token is used only
for authenticated uploads. None are returned to a technician or Proxmox host.
The broker requests only Gitea's `read:user` OAuth scope, which it uses to
confirm the signed-in username against `TAPM_ALLOWED_GITEA_USERS`.

## 4. Broker configuration

On each webserver:

```sh
git clone https://git.scity.us/TAI/TA-Deployment-Broker.git
cd TA-Deployment-Broker
cp .env.example .env
chmod 600 .env
```

Edit `.env` and use the same secrets on both webservers. The listen address is
the one node-specific value. Generate the cookie secret with:

```sh
openssl rand -base64 48
```

The database DSN must retain `parseTime=true`, `multiStatements=true`, and the
encoded UTC `time_zone` setting from `.env.example`. Its address remains
`127.0.0.1:3306`, which uses the local HAProxy frontend rather than selecting a
single Galera member.

## 5. Build and migrate

Build the image on each webserver:

```sh
docker compose build
```

Apply migrations from only one webserver:

```sh
docker compose run --rm migrate
```

Migrations use a MariaDB advisory lock, so an accidental simultaneous run
cannot apply the same migration twice.

## 6. Start both instances

On both webservers:

```sh
docker compose up -d broker
curl --fail "http://${TAPM_LISTEN_ADDR}/health/ready"
```

There is no local application state and no session affinity requirement.

For later updates, run the repository update helper from either webserver:

```sh
/opt/idssys/ta-deployment-broker/update.sh
```

The helper updates and health-checks the local broker first, then connects as
`TAPM_UPDATE_SSH_USER` to each comma-separated host in `TAPM_UPDATE_PEERS` and
performs the same local-only check there. This provides a sequential rolling
update from any node. An unreachable peer is skipped with a warning; a reachable
peer whose update fails causes the command to fail. The SSH user defaults to
`root` when omitted.

Configure each webserver's `.env` with its other broker:

```dotenv
# Webserver-Node1
TAPM_UPDATE_PEERS=10.10.1.122
TAPM_UPDATE_SSH_USER=root

# Webserver-Node2
TAPM_UPDATE_PEERS=10.10.1.121
TAPM_UPDATE_SSH_USER=root
```

Additional peers can be listed with commas. The updater reads only these named
settings and does not source `.env` as executable shell code.

Each node fetches and compares its current tracked branch with its upstream. If
both commits match, that node's Docker services are untouched. When an update
exists, the helper fast-forwards the branch, rebuilds the image, applies pending
migrations, recreates the broker container, and waits up to 150 seconds for its
Docker health check. It refuses tracked local changes and detached, untracked,
locally-ahead, or diverged branches.

## 7. HAProxy health check

The load balancer should consider an origin healthy only when
`GET /health/ready` returns HTTP 200. `/health/live` checks only the process;
`/health/ready` also verifies Galera connectivity.

Both healthy webservers can receive traffic. `leastconn` is useful because
package uploads and downloads stay open longer than portal requests.

## 8. Publish a protected package

The normal workflow is **Protected packages → Update a package** in the portal.
The package ID is the stable name used by the broker, ProxMenu, and Gitea; a
separate registry package name is not required. The broker streams the
replacement directly to Gitea, calculates SHA-256, updates the catalog, removes
the prior Gitea version, and writes audit events without buffering the installer
on local disk.

An authorization grants access to the stable package ID, not a particular
version. Existing active codes and download sessions resolve the package's
current registry version and checksum, so they automatically use a replacement
after it is registered.

The load balancer or Nginx must allow request bodies up to
`TAPM_MAX_UPLOAD_BYTES` and use a sufficiently long request timeout. The
default is 1 GiB and 30 minutes.

For recovery, a package can still be published directly:

```sh
curl --user 'publisher:PACKAGE_WRITE_TOKEN' \
  --upload-file ./SentinelAgent_linux_x86_64_v26_1_1_31.deb \
  'https://git.scity.us/api/packages/TAI/generic/sentinelone-linux/26.1.1.31/SentinelAgent_linux_x86_64_v26_1_1_31.deb'
```

Calculate its checksum:

```sh
sha256sum SentinelAgent_linux_x86_64_v26_1_1_31.deb
```

Sign in at `https://tapm.scity.us`, expand **Register package metadata
manually**, enter the matching values, and enable it. The broker never uses the
technician's Gitea credentials for package operations.

## 9. Backup and rotation

- Include `tapm_broker` in the existing Galera backup policy.
- Audit records are retained indefinitely unless an administrator establishes a
  database retention policy. The Codes view previews the newest 15 records; the
  Audit view can filter retained history by event, customer/deployment label,
  user, host, package, source IP, or details and displays up to the newest 250
  matching events.
- Keep `.env` outside Git and readable only by the service administrator.
- Rotate both Gitea package tokens and the OAuth secret if either webserver is
  compromised.
- Changing `TAPM_COOKIE_SECRET` signs out portal users; it does not invalidate
  active deployment codes.
- Revoke active deployment authorizations from the portal when a deployment
  ends early.