TAI/TA-Deployment-Broker
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

switched to fully self contained docker

Browse Source
This commit is contained in:
David Schroeder
2026-07-26 14:01:20 -05:00
parent 0ed0fb5817
commit 5d2f90ce24
34 changed files with 688 additions and 672 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deployment.md
+92 -166
View File
@@ -1,221 +1,147 @@
# Deployment
# Single-VM deployment
This service runs active-active on both webserver VMs. Both installations use
the same MariaDB Galera database, Gitea application, package registry, and
secrets. The containers use Linux host networking so `127.0.0.1:3306` reaches
each webserver's local HAProxy listener.
The production deployment lives at:
## 1. DNS and TLS
Create `tapm.scity.us` in PowerDNS and point it at the existing HAProxy
frontend. Install the certificate using the existing certificate automation.
Set the broker to listen on all addresses on both webservers:
```dotenv
TAPM_LISTEN_ADDR=0.0.0.0:8080
TAPM_DISPLAY_TIME_ZONE=America/Chicago
```text
/opt/idssys/TA-Deployment-Access
```
The wildcard bind accepts connections through each node's physical address and
through the Keepalived address `10.10.1.120` on whichever node currently owns
it. The container health check independently uses `127.0.0.1:8080`. Restrict
port 8080 to the load-balancer and trusted management addresses. If Nginx
terminates TLS locally and neither direct nor Keepalived access is required,
the broker can instead bind to `127.0.0.1:8080`.
It consists of four containers:
An example virtual host is in
[`deploy/nginx/tapm.scity.us.conf`](../deploy/nginx/tapm.scity.us.conf).
Adjust only the certificate paths if the local convention differs.
- TAPM broker, with an embedded SQLite database
- Gitea, with its own embedded SQLite database and repository storage
- Nginx, terminating TLS for the broker and Gitea
- Certbot, run on demand for certificate issuance and renewal
## 2. MariaDB Galera
Only Nginx ports 80 and 443 and Gitea SSH port 2222 are published. The broker
and Gitea HTTP services are reachable only on the private `tapm-edge` Docker
network.
Create a dedicated database and application users limited to the two webserver
addresses. Run this once against the cluster as a database administrator,
replacing the password with the same locally generated value in both accounts:
## 1. VM and network
```sql
CREATE DATABASE tapm_broker
CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci;
CREATE USER 'tapm'@'10.10.1.121' IDENTIFIED BY 'replace-with-a-random-password';
CREATE USER 'tapm'@'10.10.1.122' IDENTIFIED BY 'replace-with-a-random-password';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
ON tapm_broker.* TO 'tapm'@'10.10.1.121';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, REFERENCES
ON tapm_broker.* TO 'tapm'@'10.10.1.122';
FLUSH PRIVILEGES;
```
Install Docker Engine with Compose v2. Permit inbound TCP 80, 443, and 2222.
Forward those ports from the datacenter edge to this VM. Create public DNS A/AAAA
records for both application names before requesting the certificate.
The Keepalived address `10.10.1.120` is not included because outbound HAProxy
connections normally originate from the physical webserver address. Add a
third account for `10.10.1.120` only if a connection test or the HAProxy
configuration proves that it explicitly uses the virtual address as its source.
Port 80 must remain reachable for HTTP-01 certificate renewal. If IPv6 is
published, it must reach this same VM.
## 3. Gitea accounts and OAuth
Create these separately:
1. The `taiadmin` technician account used to sign in to the portal.
2. A `tapm-packages` service account with read access only to the private
`TAI/files` package owner. Create a personal access token for this account
and store it only in the broker environment file.
3. A `tapm-publisher` service account with `write:package` access. Its token is
used only by authenticated portal uploads.
4. An OAuth2 application with callback URL
`https://tapm.scity.us/auth/callback`.
The OAuth client secret authenticates the portal to Gitea. The read token is
used only while proxying an authorized file, and the write token is used only
for authenticated uploads. None are returned to a technician or Proxmox host.
The broker requests only Gitea's `read:user` OAuth scope, which it uses to
confirm the signed-in username against `TAPM_ALLOWED_GITEA_USERS`.
## 4. Broker configuration
On each webserver:
## 2. Install the repository
```sh
git clone https://git.scity.us/TAI/TA-Deployment-Broker.git
cd TA-Deployment-Broker
sudo mkdir -p /opt/idssys
sudo git clone GITEA-REPOSITORY-URL /opt/idssys/TA-Deployment-Access
sudo chown -R 1000:1000 /opt/idssys/TA-Deployment-Access
cd /opt/idssys/TA-Deployment-Access
cp .env.example .env
chmod 600 .env
```
Edit `.env` and use the same secrets on both webservers. The listen address is
the one node-specific value. Generate the cookie secret with:
Set `BROKER_DOMAIN`, `GITEA_DOMAIN`, and `LETSENCRYPT_EMAIL` first. Replace all
example domains in the TAPM variables. Generate the cookie secret with:
```sh
openssl rand -base64 48
```
The database DSN must retain `parseTime=true`, `multiStatements=true`, and the
encoded UTC `time_zone` setting from `.env.example`. Its address remains
`127.0.0.1:3306`, which uses the local HAProxy frontend rather than selecting a
single Galera member.
Runtime state is deliberately visible below the checkout:
## 5. Build and migrate
```text
config/
├── broker/ # tapm.db and SQLite WAL files
├── gitea/
│ ├── config/ # app.ini and Gitea configuration
│ └── data/ # repositories, packages, and gitea.db
├── letsencrypt/ # account data, certificate, and private key
├── certbot-webroot/ # HTTP-01 challenge files
└── backups/ # local offline snapshots
```
Build the image on each webserver:
The contents are ignored by Git. They must never be committed.
## 3. Bootstrap Gitea and TLS
```sh
docker compose build
./manage.sh bootstrap
```
Apply migrations from only one webserver:
This prepares the runtime directories, creates the private Docker network,
starts Gitea and the HTTP-only Nginx configuration, obtains one certificate
covering both domains, and switches Nginx to TLS.
Create the initial Gitea administrator:
```sh
docker compose run --rm migrate
docker compose --env-file .env -f deploy/gitea/compose.yaml exec gitea \
gitea admin user create \
--admin --username taiadmin --email ADMIN-EMAIL \
--password 'TEMPORARY-RANDOM-PASSWORD' --must-change-password
```
Migrations use a MariaDB advisory lock, so an accidental simultaneous run
cannot apply the same migration twice.
Sign in to Gitea at `https://GITEA_DOMAIN`, create the `TAI` organization, and
create:
## 6. Start both instances
1. `tapm-packages`, with a read-only package token.
2. `tapm-publisher`, with a write package token.
3. An OAuth2 application whose callback is
`https://BROKER_DOMAIN/auth/callback`.
On both webservers:
Place those token and OAuth values in `.env`. The broker cannot start until all
required credentials are present.
## 4. Start the complete deployment
```sh
docker compose up -d broker
curl --fail "http://127.0.0.1:8080/health/ready"
./manage.sh start
./manage.sh status
curl --fail "https://BROKER_DOMAIN/health/ready"
curl --fail "https://GITEA_DOMAIN/api/healthz"
```
There is no local application state and no session affinity requirement.
The broker automatically applies SQLite schema migrations before starting.
For later updates, run the repository update helper from either webserver:
## 5. Move repositories
For each existing repository, create an empty matching repository in the new
Gitea and mirror all refs:
```sh
/opt/idssys/ta-deployment-broker/update.sh
git clone --mirror OLD-REPOSITORY-URL
cd REPOSITORY.git
git push --mirror ssh://git@GITEA_DOMAIN:2222/TAI/REPOSITORY.git
```
The helper updates and health-checks the local broker first, then connects as
`TAPM_UPDATE_SSH_USER` to each comma-separated host in `TAPM_UPDATE_PEERS` and
performs the same local-only check there. This provides a sequential rolling
update from any node. An unreachable peer is skipped with a warning; a reachable
peer whose update fails causes the command to fail. The SSH user defaults to
`root` when omitted.
Update developer remotes, CI credentials, submodules, documentation, and the Go
module path only if the hostname embedded in the module path is also changing.
Move the deployment repository last so this checkout remains updateable during
the transition.
Configure each webserver's `.env` with its other broker:
## 6. Renewal and backups
```dotenv
# Webserver-Node1
TAPM_UPDATE_PEERS=10.10.1.122
TAPM_UPDATE_SSH_USER=root
Run renewal twice daily from root's crontab:
# Webserver-Node2
TAPM_UPDATE_PEERS=10.10.1.121
TAPM_UPDATE_SSH_USER=root
```cron
17 3,15 * * * cd /opt/idssys/TA-Deployment-Access && ./manage.sh renew
```
Additional peers can be listed with commas. The updater reads only these named
settings and does not source `.env` as executable shell code.
Each node fetches and compares its current tracked branch with its upstream. If
both commits match, that node's Docker services are untouched. When an update
exists, the helper fast-forwards the branch, rebuilds the image, applies pending
migrations, recreates the broker container, and waits up to 150 seconds for its
Docker health check. It refuses tracked local changes and detached, untracked,
locally-ahead, or diverged branches.
## 7. HAProxy health check
The load balancer should consider an origin healthy only when
`GET /health/ready` returns HTTP 200. `/health/live` checks only the process;
`/health/ready` also verifies Galera connectivity.
Both healthy webservers can receive traffic. `leastconn` is useful because
package uploads and downloads stay open longer than portal requests.
## 8. Publish a protected package
The normal workflow is **Protected packages → Update a package** in the portal.
The package ID is the stable name used by the broker, ProxMenu, and Gitea; a
separate registry package name is not required. The broker streams the
replacement directly to Gitea, calculates SHA-256, updates the catalog, removes
the prior Gitea version, and writes audit events without buffering the installer
on local disk.
An authorization grants access to the stable package ID, not a particular
version. Existing active codes and download sessions resolve the package's
current registry version and checksum, so they automatically use a replacement
after it is registered.
The load balancer or Nginx must allow request bodies up to
`TAPM_MAX_UPLOAD_BYTES` and use a sufficiently long request timeout. The
default is 1 GiB and 30 minutes.
For recovery, a package can still be published directly:
Create an application-consistent local snapshot with:
```sh
curl --user 'publisher:PACKAGE_WRITE_TOKEN' \
--upload-file ./SentinelAgent_linux_x86_64_v26_1_1_31.deb \
'https://git.scity.us/api/packages/TAI/generic/sentinelone-linux/26.1.1.31/SentinelAgent_linux_x86_64_v26_1_1_31.deb'
./manage.sh backup
```
Calculate its checksum:
The backup briefly stops both SQLite writers. Copy `config/backups/` to storage
outside the VM. A backup left only on this VM does not protect against VM or
datacenter loss. Also back up `.env` through a secrets-aware system.
## 7. Updates
```sh
sha256sum SentinelAgent_linux_x86_64_v26_1_1_31.deb
cd /opt/idssys/TA-Deployment-Access
git pull --ff-only
docker compose --env-file .env -f deploy/gitea/compose.yaml pull
./manage.sh start
```
Sign in at `https://tapm.scity.us`, expand **Register package metadata
manually**, enter the matching values, and enable it. The broker never uses the
technician's Gitea credentials for package operations.
## 9. Backup and rotation
- Include `tapm_broker` in the existing Galera backup policy.
- Audit records are retained indefinitely unless an administrator establishes a
database retention policy. The Codes view previews the newest 15 records; the
Audit view can filter retained history by event, customer/deployment label,
user, host, package, LAN/WAN IP, or details and displays up to the newest 250
matching events. Audit timestamps are stored in UTC and rendered in
`TAPM_DISPLAY_TIME_ZONE`. Deployment exchanges record both the device LAN
address reported by TAPM and the WAN/source address received through the
trusted proxy chain.
- Keep `.env` outside Git and readable only by the service administrator.
- Rotate both Gitea package tokens and the OAuth secret if either webserver is
compromised.
- Changing `TAPM_COOKIE_SECRET` signs out portal users; it does not invalidate
active deployment codes.
- Revoke active deployment authorizations from the portal when a deployment
ends early.
Pin image versions as supplied and review release notes before changing them.
Never change Gitea between rootless and rootful image families in place.