TQPro Bare-Metal Deployment Guide¶

See also: deployment-guide.md is the higher-level, environment-by-environment guide that links to this runbook. Read that first if you are trying to understand which environment you are deploying for; come back here once you know it is bare-metal (lab or AWS) and need the procedural detail. For the AWS automation proposal, see aws-cdk-automation.md.

This document provides a comprehensive, step-by-step guide for deploying TQPro on bare-metal servers to full functionality.

1. Architecture Overview¶

                        Internet
                           │
                    ┌──────┴──────┐
                    │  DNS / CDN  │
                    └──────┬──────┘
                           │
              ┌────────────┴────────────┐
              │   NGINX Gateway (SSL)   │  ← SSL offloading, HA via Keepalived
              │   Ports: 80 → 443       │
              └────────────┬────────────┘
                           │ (HTTP)
              ┌────────────┴────────────┐
              │   NGINX Web Servers     │  ← Static files + API reverse proxy
              │   3 sites: adm/pub/b2b  │
              └─────┬──────────┬────────┘
                    │          │
        ┌───────────┘          └───────────┐
        │ static files              /tlinq-api/
        │ (html-adm,               proxy_pass
        │  html-pub,                    │
        │  html-b2b)                    │
        │                    ┌──────────┴──────────┐
        │                    │  TQPro API Server    │
        │                    │  (Jetty, port 11080) │
        │                    └──────────┬──────────┘
        │                               │
        │                    ┌──────────┴──────────┐
        │                    │    PostgreSQL 16     │
        │                    │    Port 5432         │
        │                    └─────────────────────┘
        │
        │          ┌───────────────────────┐
        └─────────→│  Keycloak (OIDC)      │  ← Native OIDC auth (no oauth2-proxy)
                   │  Port 8080 (internal)  │
                   └───────────────────────┘

Components¶

Component	Purpose	Port(s)
NGINX Gateway	SSL termination, HA reverse proxy	80, 443
NGINX Web Server	Static content serving, API proxy	80
TQPro API Server	Java REST API (Jetty 12)	11080 (HTTP)
PostgreSQL	Primary database	5432
Keycloak	OIDC identity provider	8080
AWS CLI	S3 storage for CDN content and backups	—

Authentication Mode¶

TQPro uses native OIDC authentication (auth-mode=native-oidc). The frontend applications act as OIDC clients directly against Keycloak, and the API server validates JWT tokens using Keycloak's public keys (JWKS). No oauth2-proxy is required.

2. Server Prerequisites¶

2.1 Operating System¶

Ubuntu Server 22.04 LTS (recommended) or RHEL 9 / Rocky Linux 9.

2.2 System Packages¶

sudo apt update && sudo apt upgrade -y
sudo apt install -y \
    wget gnupg curl unzip rsync \
    logrotate jsvc nginx \
    apt-transport-https software-properties-common \
    ufw

2.3 Certbot (Let's Encrypt) — for Gateway / Reverse Proxy Servers¶

On servers that terminate SSL (NGINX gateway and Keycloak reverse proxy), install Certbot with the NGINX plugin:

sudo apt install -y certbot python3-certbot-nginx

Certbot manages certificate issuance, NGINX config modification, and automatic renewal via a systemd timer. Verify the renewal timer is active:

sudo systemctl status certbot.timer

Note: Certbot is only needed on front-end servers that handle SSL. Backend web/app servers and API servers do not need it.

2.4 Java 17 (Amazon Corretto) — for TQPro API Server¶

# Import Amazon Corretto GPG key
wget -O- https://apt.corretto.aws/corretto.key | \
    sudo gpg --dearmor -o /usr/share/keyrings/corretto-keyring.gpg

# Add repository
echo "deb [signed-by=/usr/share/keyrings/corretto-keyring.gpg] https://apt.corretto.aws stable main" | \
    sudo tee /etc/apt/sources.list.d/corretto.list

# Install
sudo apt update
sudo apt install -y java-17-amazon-corretto-jdk

# Verify
java -version
# Set JAVA_HOME
echo 'export JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto' | sudo tee /etc/profile.d/java.sh
source /etc/profile.d/java.sh

2.5 Java 21 (for Keycloak)¶

If Keycloak runs on the same server, install Java 21 alongside Java 17:

sudo apt install -y java-21-amazon-corretto-jdk
# Keycloak will use its own JAVA_HOME; the API server uses Java 17

If Keycloak runs on a dedicated server, install Java 21 there instead.

2.6 Python 3¶

Python is used for integration scripts (Tiqets catalog sync, utilities).

sudo apt install -y python3 python3-pip python3-venv

# Create a virtual environment for TQPro scripts
python3 -m venv /opt/tqpro-scripts/venv
source /opt/tqpro-scripts/venv/bin/activate
pip install requests>=2.28.0 psycopg2-binary>=2.9.5
deactivate

2.7 AWS CLI v2¶

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
rm -rf aws awscliv2.zip

# Verify
aws --version

Configuration — see Section 8: AWS S3 Configuration.

3. Directory Structure Setup¶

3.1 API Server Directories¶

# Create base directories
sudo mkdir -p /var/tqpro/conf
sudo mkdir -p /var/tqpro/offline-tickets
sudo mkdir -p /var/log/tqpro

# Create service user
sudo useradd -r -s /sbin/nologin -d /var/tqpro tqpro
sudo chown -R tqpro:tqpro /var/tqpro
sudo chown -R tqpro:tqpro /var/log/tqpro

Target layout:

/var/tqpro/
├── api-<BUILD_NUMBER>/       ← deployment target (build deploys here)
│   ├── tqapi.jar
│   ├── lib/                  ← dependency JARs
│   ├── libext/               ← extension JARs (optional)
│   ├── tourlinq-config.xml
│   ├── nts-client.xml
│   ├── nts-client.properties
│   ├── odoo-server.properties
│   ├── odoo-client.properties
│   ├── amadeus-client.xml
│   ├── tiqets-client.xml
│   ├── goglobal-client.xml
│   ├── googleflights-client.xml
│   ├── hazelcast.xml
│   ├── log.properties
│   ├── entities/             ← entity XML configurations
│   ├── properties.d/         ← additional properties
│   ├── tourlinq.properties   → symlink to ../conf/tourlinq.properties
│   ├── api-roles.properties  → symlink to ../conf/api-roles.properties
│   └── tlinqapi.properties   → symlink to ../conf/tlinqapi.properties
├── api/                      → symlink to latest api-<BUILD_NUMBER>/ (this is TLINQ_HOME)
├── conf/                     ← environment-specific files, NEVER overwritten by deploys
│   ├── tourlinq.properties
│   ├── api-roles.properties
│   └── tlinqapi.properties
└── offline-tickets/          ← PDF ticket storage

/var/log/tqpro/
├── tlinq-stdout.log          ← jsvc stdout
├── tlinq-stderr.log          ← jsvc stderr
└── tlinqserver-*-*.log       ← Java logging framework (auto-rotates at 10MB)

3.2 Website Directories¶

sudo mkdir -p /var/www/docimages

# Set ownership
sudo chown -R www-data:www-data /var/www

Target layout:

/var/www/
├── rel-<BUILD_NUMBER>/       ← deployment target for websites
│   ├── adm/                  ← tqweb-adm content
│   ├── pub/                  ← tqweb-pub content
│   └── b2b/                  ← tqweb-b2b content
├── html-adm/                 → symlink to latest rel-<BUILD_NUMBER>/adm
├── html-pub/                 → symlink to latest rel-<BUILD_NUMBER>/pub
├── html-b2b/                 → symlink to latest rel-<BUILD_NUMBER>/b2b
└── docimages/                ← binary file attachments (PDFs, images)

4. PostgreSQL Setup¶

4.1 Install PostgreSQL 16¶

# Add PostgreSQL APT repository
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > \
    /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt update
sudo apt install -y postgresql-16

# Enable and start
sudo systemctl enable postgresql
sudo systemctl start postgresql

4.2 Create Database and User¶

sudo -u postgres psql <<EOF
CREATE USER tlinq WITH PASSWORD '<strong-password>';
CREATE DATABASE tlinq OWNER tlinq;
GRANT ALL PRIVILEGES ON DATABASE tlinq TO tlinq;

-- Keycloak database (if Keycloak is on same server)
CREATE USER keycloak WITH PASSWORD '<keycloak-db-password>';
CREATE DATABASE keycloak OWNER keycloak;
EOF

4.3 Run Migration Scripts¶

Migration scripts are in config/db-changes/ (numbered 0001 through 0048+). Apply them in order:

# Copy scripts to server
scp config/db-changes/*.sql tqpro-server:/tmp/db-changes/

# Apply all scripts in order
for script in $(ls /tmp/db-changes/*.sql | sort); do
    echo "Applying: $script"
    sudo -u postgres psql -d tlinq -f "$script"
done

Scripts use IF EXISTS/IF NOT EXISTS guards for idempotency, so they are safe to re-run.

4.4 Configure Backups¶

Edit /var/tqpro/conf/pg_backup.conf:

PG_HOST=localhost
PG_PORT=5432
PG_USER=postgres
BACKUP_DIR=/var/backups/postgresql
RETENTION_DAYS=14
S3_UPLOAD=yes                              # Set to "yes" to upload to S3
S3_BUCKET=s3://your-bucket/pg-backups
S3_PROFILE=                                # Leave empty for IAM role on EC2
DATABASES=(
    tlinq
    keycloak
)

Set up a daily cron job:

# /etc/cron.d/tqpro-backup
0 2 * * * tqpro /var/tqpro/scripts/pg_backup.sh /var/tqpro/conf/pg_backup.conf >> /var/log/tqpro/backup.log 2>&1

5. Keycloak Setup¶

5.1 Install Keycloak (Bare-Metal)¶

# Create system user
sudo groupadd keycloak
sudo useradd -r -g keycloak -d /opt/keycloak -s /sbin/nologin keycloak

# Download and extract
KC_VERSION=26.6.1
cd /opt
sudo wget https://github.com/keycloak/keycloak/releases/download/${KC_VERSION}/keycloak-${KC_VERSION}.tar.gz
sudo tar -xzf keycloak-${KC_VERSION}.tar.gz
sudo ln -s /opt/keycloak-${KC_VERSION} /opt/keycloak
sudo chown -R keycloak:keycloak /opt/keycloak-${KC_VERSION}
sudo rm keycloak-${KC_VERSION}.tar.gz

5.2 Configure for Production¶

Edit /opt/keycloak/conf/keycloak.conf:

# Database — PostgreSQL (NEVER use H2 in production)
db=postgres
db-url=jdbc:postgresql://localhost:5432/keycloak
db-username=keycloak
db-password=<keycloak-db-password>

# Hostname (public HTTPS URL)
hostname=https://auth.yourdomain.com
hostname-strict=false

# HTTP — enabled behind reverse proxy only
http-enabled=true
http-port=8080
proxy-headers=xforwarded

# Health and metrics
health-enabled=true
metrics-enabled=true

5.3 Build and Create Admin¶

# Build (required after any config change)
sudo -u keycloak /opt/keycloak/bin/kc.sh build

# Create initial admin user
sudo -u keycloak /opt/keycloak/bin/kc.sh bootstrap-admin user \
    --username admin \
    --password <strong-admin-password>

5.4 systemd Service¶

Create /etc/systemd/system/keycloak.service:

[Unit]
Description=Keycloak Authorization Server
After=network.target postgresql.service

[Service]
Type=simple
User=keycloak
Group=keycloak
Environment=JAVA_HOME=/usr/lib/jvm/java-21-amazon-corretto
Environment=JAVA_OPTS=-Xms512m -Xmx1024m
ExecStart=/opt/keycloak/bin/kc.sh start --optimized
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable keycloak
sudo systemctl start keycloak

# Verify
curl http://localhost:9000/health/ready

5.5 Realm and Client Configuration¶

Access Keycloak admin console at https://auth.yourdomain.com
Create realm tqpro-adm
Create client tqweb-adm:
Client Protocol: openid-connect
Access Type: public (for native OIDC frontend)
Valid Redirect URIs: https://admin.yourdomain.com/*
Web Origins: https://admin.yourdomain.com
Post Logout Redirect URIs: https://admin.yourdomain.com/loggedout.html
Create roles in the realm: admin, agent, guest
Create users and assign roles
Configure token lifespans:
Access Token Lifespan: 5–15 minutes
SSO Session Idle: 30 minutes
SSO Session Max: 8–12 hours

Optional: Create tqpro-web realm for public site authentication (B2C/B2B).

5.6 NGINX Reverse Proxy for Keycloak¶

Step 1: Create the initial HTTP-only stub at /etc/nginx/sites-available/keycloak.conf:

server {
    listen 80;
    server_name auth.yourdomain.com;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Enable and test:

sudo ln -s /etc/nginx/sites-available/keycloak.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

Step 2: Run Certbot to obtain the SSL certificate and automatically update the config:

sudo certbot --nginx -d auth.yourdomain.com

Certbot will: - Obtain a Let's Encrypt certificate for auth.yourdomain.com - Modify keycloak.conf to add listen 443 ssl, certificate paths, and an HTTP→HTTPS redirect - Set up automatic renewal via systemd timer

Verify the result:

# Check the modified config
cat /etc/nginx/sites-available/keycloak.conf

# Test renewal
sudo certbot renew --dry-run

6. NGINX Configuration¶

TQPro uses a two-tier NGINX architecture:

Gateway servers — SSL offloading, HA with Keepalived, proxy to web servers
Web/App servers — serve static content, proxy /tlinq-api/ to API servers

6.1 Shared Upstream Definitions¶

Create on both gateway and web servers.

Gateway Upstreams — `/etc/nginx/conf.d/upstreams.conf`¶

# Web server upstreams (gateway → web servers)
upstream backend_adm {
    server web01.internal:80;
    server web02.internal:80;
}

upstream backend_pub {
    server web01.internal:80;
    server web02.internal:80;
}

upstream backend_b2b {
    server web01.internal:80;
    server web02.internal:80;
}

Web Server Upstreams — `/etc/nginx/conf.d/upstreams.conf`¶

# API server upstream (web servers → API servers)
upstream tqapi_backend {
    server api01.internal:11080;
    server api02.internal:11080;
}

6.2 Gateway Server Configuration¶

Install Keepalived for HA:

sudo apt install -y keepalived

Configure /etc/keepalived/keepalived.conf on primary gateway:

vrrp_instance VI_1 {
    state MASTER
    interface ens18
    virtual_router_id 51
    priority 101
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass <shared-secret>
    }

    virtual_ipaddress {
        <floating-ip>
    }
}

On secondary gateway, change state BACKUP and priority 100.

sudo systemctl enable keepalived
sudo systemctl start keepalived

Step 1: Create HTTP-Only Stub Configs¶

Start with plain HTTP configs. Certbot will add SSL configuration automatically.

Admin Site — /etc/nginx/sites-available/tqweb-adm-gw.conf

server {
    listen 80;
    server_name admin.yourdomain.com;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    location / {
        proxy_pass http://backend_adm;
    }
}

Public Site — /etc/nginx/sites-available/tqweb-pub-gw.conf

server {
    listen 80;
    server_name www.yourdomain.com;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    location / {
        proxy_pass http://backend_pub;
    }
}

B2B Site — /etc/nginx/sites-available/tqweb-b2b-gw.conf

server {
    listen 80;
    server_name b2b.yourdomain.com;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    location / {
        proxy_pass http://backend_b2b;
    }
}

Enable all gateway sites:

sudo ln -s /etc/nginx/sites-available/tqweb-adm-gw.conf /etc/nginx/sites-enabled/
sudo ln -s /etc/nginx/sites-available/tqweb-pub-gw.conf /etc/nginx/sites-enabled/
sudo ln -s /etc/nginx/sites-available/tqweb-b2b-gw.conf /etc/nginx/sites-enabled/
sudo rm -f /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

Step 2: Obtain SSL Certificates with Certbot¶

Run Certbot for each site. Certbot will automatically modify the config files to add listen 443 ssl, certificate paths, SSL parameters, and an HTTP→HTTPS redirect.

# Admin site
sudo certbot --nginx -d admin.yourdomain.com

# Public site
sudo certbot --nginx -d www.yourdomain.com

# B2B site
sudo certbot --nginx -d b2b.yourdomain.com

Or obtain all certificates in a single command (uses one multi-domain certificate):

sudo certbot --nginx \
    -d admin.yourdomain.com \
    -d www.yourdomain.com \
    -d b2b.yourdomain.com

After Certbot completes, each config file will have been updated with SSL blocks. Certbot also adds HSTS headers and HTTP→HTTPS redirects.

Step 3: Verify SSL and Auto-Renewal¶

# Check that the modified configs are valid
sudo nginx -t

# Verify certificates
sudo certbot certificates

# Test auto-renewal (dry run)
sudo certbot renew --dry-run

# Verify the renewal timer is active
sudo systemctl status certbot.timer

Certbot's systemd timer runs twice daily and renews certificates that are within 30 days of expiry. No manual intervention is needed after initial setup.

6.3 Web/App Server Configuration¶

Admin Site — `/etc/nginx/sites-available/tqweb-adm.conf`¶

server {
    listen 80;
    server_name admin.yourdomain.com;

    root /var/www/html-adm;
    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }

    location /tlinq-api/ {
        proxy_pass http://tqapi_backend/tlinq-api/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 10;
        proxy_read_timeout 60;
    }

    location /docimages/ {
        alias /var/www/docimages/;
    }
}

Public Site — `/etc/nginx/sites-available/tqweb-pub.conf`¶

server {
    listen 80;
    server_name www.yourdomain.com;

    root /var/www/html-pub;
    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }

    location /tlinq-api/ {
        proxy_pass http://tqapi_backend/tlinq-api/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 10;
        proxy_read_timeout 60;
    }

    location /docimages/ {
        alias /var/www/docimages/;
    }
}

B2B Site — `/etc/nginx/sites-available/tqweb-b2b.conf`¶

server {
    listen 80;
    server_name b2b.yourdomain.com;

    root /var/www/html-b2b;
    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }

    location /tlinq-api/ {
        proxy_pass http://tqapi_backend/tlinq-api/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 10;
        proxy_read_timeout 60;
    }

    location /docimages/ {
        alias /var/www/docimages/;
    }
}

Enable all web server sites:

sudo ln -s /etc/nginx/sites-available/tqweb-adm.conf /etc/nginx/sites-enabled/
sudo ln -s /etc/nginx/sites-available/tqweb-pub.conf /etc/nginx/sites-enabled/
sudo ln -s /etc/nginx/sites-available/tqweb-b2b.conf /etc/nginx/sites-enabled/
sudo rm /etc/nginx/sites-enabled/default  # Remove default site
sudo nginx -t && sudo systemctl reload nginx

7. AWS S3 Configuration¶

7.1 Non-EC2 VM (Local / On-Premises)¶

aws configure
# AWS Access Key ID: <your-access-key>
# AWS Secret Access Key: <your-secret-key>
# Default region name: <your-region>  (e.g., me-south-1)
# Default output format: json

Credentials are stored in ~/.aws/credentials. For the tqpro service user:

sudo -u tqpro aws configure

7.2 EC2 VM (AWS)¶

On EC2 instances, use an IAM Instance Role — no credentials file needed. Attach a role with the following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

7.3 Configure CDN Prefix¶

In tourlinq.properties, set the CloudFront CDN prefix:

content.cdn-prefix=https://d1234abcdef.cloudfront.net/

8. API Server Deployment¶

8.1 Build the API Server¶

On the build server:

cd /path/to/tqpro
./gradlew clean build copyDependencies

Build outputs: - tqapi/build/libs/tqapi.jar — main application JAR - tqapi/build/libs/lib/ — dependency JARs (after copyDependencies)

8.2 Deploy Script — `deploy-api.sh`¶

Note: This script is provided for illustration and manual/emergency deployments only. In the normal SDLC workflow, the TeamCity build project contains the equivalent deployment steps (build, upload artifacts, deploy to target, swap symlinks, restart service). You do not need this script if your CI/CD pipeline is in place. See doc/operations/build-and-ci/build-strategy.md for the TeamCity pipeline configuration.

#!/bin/bash
#
# TQPro API Server Deployment Script (illustrative — see note above)
# Usage: ./deploy-api.sh <BUILD_NUMBER> <BUILD_ARTIFACTS_DIR>
#

set -euo pipefail

BUILD_NUMBER="${1:?Usage: $0 <BUILD_NUMBER> <BUILD_ARTIFACTS_DIR>}"
ARTIFACTS_DIR="${2:?Usage: $0 <BUILD_NUMBER> <BUILD_ARTIFACTS_DIR>}"
TQPRO_BASE="/var/tqpro"
DEPLOY_DIR="${TQPRO_BASE}/api-${BUILD_NUMBER}"
CONF_DIR="${TQPRO_BASE}/conf"
SERVICE_USER="tqpro"

echo "=== Deploying TQPro API build #${BUILD_NUMBER} ==="

# 1. Create deployment directory
if [ -d "$DEPLOY_DIR" ]; then
    echo "ERROR: Deploy directory already exists: $DEPLOY_DIR"
    exit 1
fi
sudo -u $SERVICE_USER mkdir -p "$DEPLOY_DIR"/{lib,libext,entities,properties.d,tmp}

# 2. Copy build artifacts
sudo -u $SERVICE_USER cp "$ARTIFACTS_DIR/tqapi.jar" "$DEPLOY_DIR/"
sudo -u $SERVICE_USER cp "$ARTIFACTS_DIR/lib/"*.jar "$DEPLOY_DIR/lib/"
if ls "$ARTIFACTS_DIR/libext/"*.jar 1>/dev/null 2>&1; then
    sudo -u $SERVICE_USER cp "$ARTIFACTS_DIR/libext/"*.jar "$DEPLOY_DIR/libext/"
fi

# 3. Copy configuration files (non-environment-specific)
CONFIG_SRC="/path/to/tqpro/config"
sudo -u $SERVICE_USER cp "$CONFIG_SRC/tourlinq-config.xml" "$DEPLOY_DIR/"
sudo -u $SERVICE_USER cp "$CONFIG_SRC/nts-client.xml" "$DEPLOY_DIR/"
sudo -u $SERVICE_USER cp "$CONFIG_SRC/nts-client.properties" "$DEPLOY_DIR/"
sudo -u $SERVICE_USER cp "$CONFIG_SRC/odoo-server.properties" "$DEPLOY_DIR/"
sudo -u $SERVICE_USER cp "$CONFIG_SRC/odoo-client.properties" "$DEPLOY_DIR/"
sudo -u $SERVICE_USER cp "$CONFIG_SRC/amadeus-client.xml" "$DEPLOY_DIR/"
sudo -u $SERVICE_USER cp "$CONFIG_SRC/tiqets-client.xml" "$DEPLOY_DIR/"
sudo -u $SERVICE_USER cp "$CONFIG_SRC/goglobal-client.xml" "$DEPLOY_DIR/"
sudo -u $SERVICE_USER cp "$CONFIG_SRC/googleflights-client.xml" "$DEPLOY_DIR/"
sudo -u $SERVICE_USER cp "$CONFIG_SRC/hazelcast.xml" "$DEPLOY_DIR/"
sudo -u $SERVICE_USER cp "$CONFIG_SRC/log.properties" "$DEPLOY_DIR/"
sudo -u $SERVICE_USER cp -r "$CONFIG_SRC/entities/" "$DEPLOY_DIR/entities/"
sudo -u $SERVICE_USER cp "$CONFIG_SRC/properties.d/"*.properties "$DEPLOY_DIR/properties.d/" 2>/dev/null || true

# 4. Copy service scripts
sudo -u $SERVICE_USER cp "$CONFIG_SRC/tlinq-service.sh" "$DEPLOY_DIR/"
sudo -u $SERVICE_USER chmod +x "$DEPLOY_DIR/tlinq-service.sh"

# 5. Create symlinks to environment-specific configs
cd "$DEPLOY_DIR"
sudo -u $SERVICE_USER ln -sf ../conf/tourlinq.properties tourlinq.properties
sudo -u $SERVICE_USER ln -sf ../conf/api-roles.properties api-roles.properties
sudo -u $SERVICE_USER ln -sf ../conf/tlinqapi.properties tlinqapi.properties

# 6. Stop current server (if running)
CURRENT_LINK=$(readlink -f "${TQPRO_BASE}/api" 2>/dev/null || echo "")
if [ -n "$CURRENT_LINK" ] && [ -f "$CURRENT_LINK/tlinq-service.sh" ]; then
    echo "Stopping current server..."
    cd "$CURRENT_LINK"
    ./tlinq-service.sh stop || echo "WARNING: Could not stop current server"
    sleep 2
fi

# 7. Atomic symlink swap
cd "$TQPRO_BASE"
sudo -u $SERVICE_USER ln -sfn "api-${BUILD_NUMBER}" api

# 8. Start new server
cd "${TQPRO_BASE}/api"
./tlinq-service.sh start

echo "=== Deployment complete: build #${BUILD_NUMBER} ==="
echo "TLINQ_HOME: ${TQPRO_BASE}/api"
echo "Actual path: ${DEPLOY_DIR}"

8.3 TLINQ_HOME Contents After Deployment¶

/var/tqpro/api/ (→ api-<BUILD_NUMBER>)
├── tqapi.jar                     ← Main application JAR
├── lib/*.jar                     ← Dependency JARs
├── libext/*.jar                  ← Extension JARs (optional)
├── tlinq-service.sh              ← Service management script
├── tourlinq-config.xml           ← Master plugin/entity config
├── nts-client.xml                ← NTS plugin services (78KB)
├── nts-client.properties         ← NTS connection settings
├── odoo-server.properties        ← Odoo ERP connection
├── odoo-client.properties        ← Odoo service mappings
├── amadeus-client.xml            ← Amadeus GDS config
├── tiqets-client.xml             ← Tiqets API config
├── goglobal-client.xml           ← GoGlobal API config
├── googleflights-client.xml      ← Google Flights config
├── hazelcast.xml                 ← Distributed cache config
├── log.properties                ← Java logging config
├── entities/                     ← 21 domain-specific entity XMLs
├── properties.d/
│   └── erp-booking.properties    ← Payment/ERP mappings
├── tmp/                          ← Multipart upload temp dir
├── tourlinq.properties           → ../conf/tourlinq.properties
├── api-roles.properties          → ../conf/api-roles.properties
└── tlinqapi.properties           → ../conf/tlinqapi.properties

9. Website Deployment¶

9.1 Deploy Script — `deploy-web.sh`¶

Note: As with the API deploy script, this is for illustration and manual/emergency deployments only. The TeamCity build pipeline handles website deployment (rsync to target, symlink swap) as part of the normal SDLC workflow.

#!/bin/bash
#
# TQPro Website Deployment Script (illustrative — see note above)
# Usage: ./deploy-web.sh <BUILD_NUMBER> <WEB_ARTIFACTS_DIR>
#

set -euo pipefail

BUILD_NUMBER="${1:?Usage: $0 <BUILD_NUMBER> <WEB_ARTIFACTS_DIR>}"
ARTIFACTS_DIR="${2:?Usage: $0 <BUILD_NUMBER> <WEB_ARTIFACTS_DIR>}"
WWW_BASE="/var/www"
DEPLOY_DIR="${WWW_BASE}/rel-${BUILD_NUMBER}"

echo "=== Deploying TQPro websites build #${BUILD_NUMBER} ==="

# 1. Create deployment directory
if [ -d "$DEPLOY_DIR" ]; then
    echo "ERROR: Deploy directory already exists: $DEPLOY_DIR"
    exit 1
fi
sudo mkdir -p "$DEPLOY_DIR"/{adm,pub,b2b}

# 2. Copy website content
sudo rsync -a "$ARTIFACTS_DIR/tqweb-adm/" "$DEPLOY_DIR/adm/"
sudo rsync -a "$ARTIFACTS_DIR/tqweb-pub/" "$DEPLOY_DIR/pub/"
sudo rsync -a "$ARTIFACTS_DIR/tqweb-b2b/" "$DEPLOY_DIR/b2b/"

# 3. Set permissions
sudo chown -R www-data:www-data "$DEPLOY_DIR"

# 4. Atomic symlink swap
cd "$WWW_BASE"
sudo ln -sfn "rel-${BUILD_NUMBER}/adm" html-adm
sudo ln -sfn "rel-${BUILD_NUMBER}/pub" html-pub
sudo ln -sfn "rel-${BUILD_NUMBER}/b2b" html-b2b

echo "=== Website deployment complete: build #${BUILD_NUMBER} ==="
echo "  Admin: /var/www/html-adm → rel-${BUILD_NUMBER}/adm"
echo "  Public: /var/www/html-pub → rel-${BUILD_NUMBER}/pub"
echo "  B2B: /var/www/html-b2b → rel-${BUILD_NUMBER}/b2b"

9.2 Rollback¶

To rollback to a previous build, simply re-point the symlinks:

# Rollback API
cd /var/tqpro
sudo -u tqpro ln -sfn api-<PREVIOUS_BUILD> api
cd /var/tqpro/api
./tlinq-service.sh restart

# Rollback websites
cd /var/www
sudo ln -sfn rel-<PREVIOUS_BUILD>/adm html-adm
sudo ln -sfn rel-<PREVIOUS_BUILD>/pub html-pub
sudo ln -sfn rel-<PREVIOUS_BUILD>/b2b html-b2b

10. Environment-Specific Configuration Checklist¶

These files live in /var/tqpro/conf/ and are never overwritten by deployments. Set them up once per environment and update as needed.

10.1 `tlinqapi.properties`¶

Property	Description	Example
`http-port`	API HTTP port	`11080`
`https-port`	API HTTPS port	`11079`
`http-enabled`	Enable HTTP (true if behind proxy)	`true`
`key-file`	HTTPS keystore file	`perun.jks`
`key-pass`	Keystore password	(externalize in production)
`content-location`	Binary attachments directory	`/var/www/docimages`
`auth-mode`	Authentication mode	`native-oidc`
`oidc-issuer`	Keycloak realm issuer URL	`https://auth.yourdomain.com/realms/tqpro-adm`
`oidc-client-id`	OIDC client ID	`tqweb-adm`
`oidc-roles-claim`	JWT claim path for roles	`realm_access.roles`
`oidc-post-logout-redirect-uri`	Post-logout URL	`https://admin.yourdomain.com/loggedout.html`
`oidc-jwks-cache-lifetime`	JWKS cache seconds	`300`
`cors-allowed-origins`	CORS origins (comma-separated)	`*.yourdomain.com,localhost`
`dev-mode`	MUST be `false` in production	`false`

10.2 `tourlinq.properties`¶

Property	Description	Example
`company.code`	Company identifier	`C0002`
`tlinq.dbname`	Database environment name	`prod`
`tlinq.dbpass`	Database password	(use strong password)
`mail.server`	SMTP server	`smtp.zoho.com`
`mail.port`	SMTP port	`587`
`mail.user`	SMTP username	`noreply@yourdomain.com`
`mail.password`	SMTP password	(externalize)
`mail.from`	From address	`noreply@yourdomain.com`
`mail.usetls`	Enable TLS	`true`
`content.directory`	Content directory	`/var/www/html/content`
`content.cdn-prefix`	CloudFront CDN URL	`https://d1234.cloudfront.net/`
`pgw.class`	Payment gateway class	`com.perun.tlinq.pgw.telrj.TelrJPgw`
`telr.pgw-url`	Telr gateway URL	`https://secure.telr.com/gateway/order.json`
`telr.auth-key`	Telr auth key	(from Telr dashboard)
`telr.store-id`	Telr store ID	(from Telr dashboard)
`telr.test-mode`	Telr test mode	`0` (production) / `1` (test)
`telr.link-success`	Payment success URL	`https://www.yourdomain.com/paymentsuccess.html`
`telr.link-cancel`	Payment cancel URL	`https://www.yourdomain.com/paymentcanc.html`
`telr.link-declined`	Payment declined URL	`https://www.yourdomain.com/paymentdecl.html`
`pgw.callback-base`	Payment callback base URL	`https://www.yourdomain.com/tlinq-api`
`pgw.public-site-url`	Public site URL	`https://www.yourdomain.com`
`tiqets.api.key`	Tiqets API key	(from Tiqets)
`goglobal.api.password`	GoGlobal API password	(from GoGlobal)
`rapidapi.visa.key`	RapidAPI visa key	(from RapidAPI)
`gf.rapidapi.key`	Google Flights RapidAPI key	(from RapidAPI)
`twilio.sid`	Twilio account SID	(from Twilio)
`twilio.token`	Twilio auth token	(from Twilio)
`twilio.sms.sender`	Twilio SMS sender ID	(from Twilio)
`twilio.wa.sender`	Twilio WhatsApp sender ID	(from Twilio)
`amadeus.idfile`	Amadeus credentials file path	`/var/tqpro/conf/amadeus.idfile`
`ai.api.key`	Claude/Anthropic API key	(from Anthropic)
`ai.model`	AI model name	`claude-sonnet-4-5-20250929`
`pexels.api.key`	Pexels API key	(from Pexels)
`offline-tickets.storage-path`	Offline ticket PDF storage	`/var/tqpro/offline-tickets`
`company.support-email`	Support email address	`reservations@yourdomain.com`

10.3 `api-roles.properties`¶

Copy from config/api-roles.properties as a baseline. This file maps API endpoints to required roles (guest, agent, admin). Review and customize per environment.

10.4 Other Configuration Files¶

These are deployed with the build but may need environment-specific edits:

File	Key Settings to Review
`tourlinq-config.xml`	Database environment entries (JDBC URLs, usernames), enabled plugins/factories
`odoo-server.properties`	Odoo server URL, database name, credentials
`hazelcast.xml`	Cluster member IPs (update for each environment)
`properties.d/erp-booking.properties`	Payment channel mappings, service charge rates, ERP product codes
`log.properties`	Log file path (update to `/var/log/tqpro/tlinqserver-%g-%u.log`)

11. Service Management¶

11.1 TQPro API Service (jsvc)¶

The tlinq-service.sh script provides production-grade service management:

cd /var/tqpro/api

# Start the server
./tlinq-service.sh start

# Stop the server
./tlinq-service.sh stop

# Restart
./tlinq-service.sh restart

# Check status (PID, uptime, memory)
./tlinq-service.sh status

# Rotate logs WITHOUT restart
./tlinq-service.sh rotate

# Tail logs
./tlinq-service.sh logs stderr    # Error output
./tlinq-service.sh logs stdout    # Standard output
./tlinq-service.sh logs java      # Java logging framework

11.2 systemd Service for TQPro API¶

Create /etc/systemd/system/tqpro.service:

[Unit]
Description=TQPro API Server
After=network.target postgresql.service

[Service]
Type=forking
User=tqpro
WorkingDirectory=/var/tqpro/api
ExecStart=/var/tqpro/api/tlinq-service.sh start
ExecStop=/var/tqpro/api/tlinq-service.sh stop
ExecReload=/var/tqpro/api/tlinq-service.sh rotate
PIDFile=/var/tqpro/api/tlinq.pid
Restart=on-failure
RestartSec=10
Environment=JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable tqpro
sudo systemctl start tqpro

11.3 Debug Mode¶

To enable remote debugging (port 5005), create the debug indicator file:

touch /var/tqpro/conf/debug.ind

Remove it and restart to disable debug mode.

11.4 Log Rotation¶

Install the logrotate configuration:

sudo cp logrotate-tlinq.conf /etc/logrotate.d/tqpro

Edit /etc/logrotate.d/tqpro to set correct user/group and paths:

/var/log/tqpro/tlinq-stdout.log /var/log/tqpro/tlinq-stderr.log {
    daily
    size 100M
    rotate 30
    missingok
    notifempty
    compress
    delaycompress
    create 0644 tqpro tqpro
    dateext
    dateformat -%Y%m%d-%s
    postrotate
        TLINQ_HOME="/var/tqpro/api"
        if [ -f "$TLINQ_HOME/tlinq.pid" ]; then
            kill -HUP $(cat "$TLINQ_HOME/tlinq.pid") 2>/dev/null || true
        fi
    endscript
    sharedscripts
}

/var/log/tqpro/tlinqserver-*.log {
    compress
    maxage 60
    missingok
    nocreate
    norotate
}

Test: sudo logrotate -d /etc/logrotate.d/tqpro

12. Firewall Configuration¶

# Allow SSH
sudo ufw allow ssh

# Allow HTTP and HTTPS (gateway servers only)
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

# Allow API port from web servers only (API servers)
sudo ufw allow from <web-server-1-ip> to any port 11080
sudo ufw allow from <web-server-2-ip> to any port 11080

# Allow PostgreSQL from API servers only (database server)
sudo ufw allow from <api-server-1-ip> to any port 5432
sudo ufw allow from <api-server-2-ip> to any port 5432

# Allow Keycloak from gateway/web servers (Keycloak server)
sudo ufw allow from <gateway-ip> to any port 8080

# Enable firewall
sudo ufw enable

13. Post-Deployment Verification¶

13.1 Service Health Checks¶

# TQPro API Server
curl -s http://localhost:11080/tlinq-api/ping
# Expected: API response with status

# Keycloak
curl -s http://localhost:8080/health/ready
# Expected: {"status":"UP"}

# NGINX
sudo nginx -t
sudo systemctl status nginx

# PostgreSQL
sudo -u postgres pg_isready

13.2 End-to-End Test¶

Open https://admin.yourdomain.com in a browser
Verify redirect to Keycloak login page
Login with test credentials
Verify the admin dashboard loads
Test an API call (e.g., list customers)

13.3 API Test with JWT¶

# Get a token from Keycloak
TOKEN=$(curl -s -X POST \
    "https://auth.yourdomain.com/realms/tqpro-adm/protocol/openid-connect/token" \
    -d "grant_type=password" \
    -d "client_id=tqweb-adm" \
    -d "username=testuser" \
    -d "password=testpass" | jq -r '.access_token')

# Test API endpoint
curl -s -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"session":""}' \
    http://localhost:11080/tlinq-api/customer/list

14. Production Checklist¶

Security¶

[ ] dev-mode=false in tlinqapi.properties
[ ] auth-mode=native-oidc configured with correct Keycloak URLs
[ ] SSL certificates installed and valid (not self-signed)
[ ] Keystore password externalized (not default PerunKeyPass)
[ ] Database passwords are strong and unique per environment
[ ] API keys and secrets in tourlinq.properties are production values
[ ] telr.test-mode=0 for production payment processing
[ ] Firewall rules restrict internal ports (11080, 5432, 8080)
[ ] NGINX server_tokens off to hide version

Infrastructure¶

[ ] PostgreSQL database created with migrations applied
[ ] Keycloak realm, client, roles, and users configured
[ ] NGINX gateway with SSL configured for all three sites
[ ] NGINX web servers configured with API proxy
[ ] DNS records pointing to gateway floating IP
[ ] Keepalived HA configured (if using multiple gateways)
[ ] AWS CLI configured (credentials or IAM role)

Deployment¶

[ ] /var/tqpro/conf/ populated with environment-specific properties
[ ] API server deployed and running (tlinq-service.sh status)
[ ] Websites deployed and symlinks created
[ ] systemd services enabled (tqpro, keycloak, nginx, postgresql)
[ ] Log rotation configured (/etc/logrotate.d/tqpro)
[ ] Log directory permissions correct (/var/log/tqpro/)

Monitoring & Maintenance¶

[ ] PostgreSQL backup cron job configured
[ ] Keycloak realm export procedure documented
[ ] Disk space monitoring for /var/log/tqpro/ and /var/backups/
[ ] SSL certificate renewal automated (Let's Encrypt) or calendar reminder
[ ] Old deployment directories cleanup procedure in place
[ ] OIDC JWKS discovery endpoint reachable from API servers