David Schroeder
189 lines
6.8 KiB
Markdown
189 lines
6.8 KiB
Markdown
# Deployment
|
|
|
|
This service runs active-active on both webserver VMs. Both installations use
|
|
the same MariaDB Galera database, Gitea application, package registry, and
|
|
secrets. The containers use Linux host networking so `127.0.0.1:3306` reaches
|
|
each webserver's local HAProxy listener.
|
|
|
|
## 1. DNS and TLS
|
|
|
|
Create `tapm.scity.us` in PowerDNS and point it at the existing HAProxy
|
|
frontend. Install the certificate using the existing certificate automation.
|
|
Set `TAPM_LISTEN_ADDR` to each node's physical address when the load balancer
|
|
connects directly to port 8080:
|
|
|
|
- Webserver-Node1: `10.10.1.121:8080`
|
|
- Webserver-Node2: `10.10.1.122:8080`
|
|
|
|
Restrict port 8080 to the load-balancer addresses. If Nginx terminates TLS on
|
|
each webserver instead, keep the broker on `127.0.0.1:8080`.
|
|
|
|
An example virtual host is in
|
|
[`deploy/nginx/tapm.scity.us.conf`](../deploy/nginx/tapm.scity.us.conf).
|
|
Adjust only the certificate paths if the local convention differs.
|
|
|
|
## 2. MariaDB Galera
|
|
|
|
Create a dedicated database and application users limited to the two webserver
|
|
addresses. Run this once against the cluster as a database administrator,
|
|
replacing the password with the same locally generated value in both accounts:
|
|
|
|
```sql
|
|
CREATE DATABASE tapm_broker
|
|
CHARACTER SET utf8mb4
|
|
COLLATE utf8mb4_unicode_ci;
|
|
CREATE USER 'tapm'@'10.10.1.121' IDENTIFIED BY 'replace-with-a-random-password';
|
|
CREATE USER 'tapm'@'10.10.1.122' IDENTIFIED BY 'replace-with-a-random-password';
|
|
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
|
|
ON tapm_broker.* TO 'tapm'@'10.10.1.121';
|
|
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
|
|
ON tapm_broker.* TO 'tapm'@'10.10.1.122';
|
|
FLUSH PRIVILEGES;
|
|
```
|
|
|
|
The Keepalived address `10.10.1.120` is not included because outbound HAProxy
|
|
connections normally originate from the physical webserver address. Add a
|
|
third account for `10.10.1.120` only if a connection test or the HAProxy
|
|
configuration proves that it explicitly uses the virtual address as its source.
|
|
|
|
## 3. Gitea accounts and OAuth
|
|
|
|
Create these separately:
|
|
|
|
1. The `taiadmin` technician account used to sign in to the portal.
|
|
2. A `tapm-packages` service account with read access only to the private
|
|
`TAI/files` package owner. Create a personal access token for this account
|
|
and store it only in the broker environment file.
|
|
3. A `tapm-publisher` service account with `write:package` access. Its token is
|
|
used only by authenticated portal uploads.
|
|
4. An OAuth2 application with callback URL
|
|
`https://tapm.scity.us/auth/callback`.
|
|
|
|
The OAuth client secret authenticates the portal to Gitea. The read token is
|
|
used only while proxying an authorized file, and the write token is used only
|
|
for authenticated uploads. None are returned to a technician or Proxmox host.
|
|
The broker requests only Gitea's `read:user` OAuth scope, which it uses to
|
|
confirm the signed-in username against `TAPM_ALLOWED_GITEA_USERS`.
|
|
|
|
## 4. Broker configuration
|
|
|
|
On each webserver:
|
|
|
|
```sh
|
|
git clone https://git.scity.us/TAI/TA-Deployment-Broker.git
|
|
cd TA-Deployment-Broker
|
|
cp .env.example .env
|
|
chmod 600 .env
|
|
```
|
|
|
|
Edit `.env` and use the same secrets on both webservers. The listen address is
|
|
the one node-specific value. Generate the cookie secret with:
|
|
|
|
```sh
|
|
openssl rand -base64 48
|
|
```
|
|
|
|
The database DSN must retain `parseTime=true`, `multiStatements=true`, and the
|
|
encoded UTC `time_zone` setting from `.env.example`. Its address remains
|
|
`127.0.0.1:3306`, which uses the local HAProxy frontend rather than selecting a
|
|
single Galera member.
|
|
|
|
## 5. Build and migrate
|
|
|
|
Build the image on each webserver:
|
|
|
|
```sh
|
|
docker compose build
|
|
```
|
|
|
|
Apply migrations from only one webserver:
|
|
|
|
```sh
|
|
docker compose run --rm migrate
|
|
```
|
|
|
|
Migrations use a MariaDB advisory lock, so an accidental simultaneous run
|
|
cannot apply the same migration twice.
|
|
|
|
## 6. Start both instances
|
|
|
|
On both webservers:
|
|
|
|
```sh
|
|
docker compose up -d broker
|
|
curl --fail "http://${TAPM_LISTEN_ADDR}/health/ready"
|
|
```
|
|
|
|
There is no local application state and no session affinity requirement.
|
|
|
|
For later updates, run the repository update helper on one webserver at a time:
|
|
|
|
```sh
|
|
/opt/idssys/ta-deployment-broker/update.sh
|
|
```
|
|
|
|
The helper fast-forwards the current tracked branch, rebuilds the image, applies
|
|
any pending migrations, recreates the broker container, and waits up to 150
|
|
seconds for its Docker health check. It refuses to pull over tracked local
|
|
changes or from a detached/untracked branch.
|
|
|
|
## 7. HAProxy health check
|
|
|
|
The load balancer should consider an origin healthy only when
|
|
`GET /health/ready` returns HTTP 200. `/health/live` checks only the process;
|
|
`/health/ready` also verifies Galera connectivity.
|
|
|
|
Both healthy webservers can receive traffic. `leastconn` is useful because
|
|
package uploads and downloads stay open longer than portal requests.
|
|
|
|
## 8. Publish a protected package
|
|
|
|
The normal workflow is **Protected packages → Update a package** in the portal.
|
|
The package ID is the stable name used by the broker, ProxMenu, and Gitea; a
|
|
separate registry package name is not required. The broker streams the
|
|
replacement directly to Gitea, calculates SHA-256, updates the catalog, removes
|
|
the prior Gitea version, and writes audit events without buffering the installer
|
|
on local disk.
|
|
|
|
An authorization grants access to the stable package ID, not a particular
|
|
version. Existing active codes and download sessions resolve the package's
|
|
current registry version and checksum, so they automatically use a replacement
|
|
after it is registered.
|
|
|
|
The load balancer or Nginx must allow request bodies up to
|
|
`TAPM_MAX_UPLOAD_BYTES` and use a sufficiently long request timeout. The
|
|
default is 1 GiB and 30 minutes.
|
|
|
|
For recovery, a package can still be published directly:
|
|
|
|
```sh
|
|
curl --user 'publisher:PACKAGE_WRITE_TOKEN' \
|
|
--upload-file ./SentinelAgent_linux_x86_64_v26_1_1_31.deb \
|
|
'https://git.scity.us/api/packages/TAI/generic/sentinelone-linux/26.1.1.31/SentinelAgent_linux_x86_64_v26_1_1_31.deb'
|
|
```
|
|
|
|
Calculate its checksum:
|
|
|
|
```sh
|
|
sha256sum SentinelAgent_linux_x86_64_v26_1_1_31.deb
|
|
```
|
|
|
|
Sign in at `https://tapm.scity.us`, expand **Register package metadata
|
|
manually**, enter the matching values, and enable it. The broker never uses the
|
|
technician's Gitea credentials for package operations.
|
|
|
|
## 9. Backup and rotation
|
|
|
|
- Include `tapm_broker` in the existing Galera backup policy.
|
|
- Audit records are retained indefinitely unless an administrator establishes a
|
|
database retention policy. The Codes view previews the newest 15 records; the
|
|
Audit view can filter the retained history and displays up to the newest 250
|
|
matching events.
|
|
- Keep `.env` outside Git and readable only by the service administrator.
|
|
- Rotate both Gitea package tokens and the OAuth secret if either webserver is
|
|
compromised.
|
|
- Changing `TAPM_COOKIE_SECRET` signs out portal users; it does not invalidate
|
|
active deployment codes.
|
|
- Revoke active deployment authorizations from the portal when a deployment
|
|
ends early.
|