How to install GPT Gateway (old)

1 GPT Gateway Fresh Install Guide

GPT Gateway Fresh Install Guide

If you’re deploying private AI please refer to this wiki https://agatsoftware.atlassian.net/wiki/spaces/VA/pages/3200122882

Use the same image of LLM for the Gateway

If you already have existing environment, please use this tutorial: https://agatsoftware.atlassian.net/wiki/spaces/SKYP/pages/3098050611

Setting Up a Hostname:
- Create a hostname for the GPT gateway API.
- Example: https://bgdXgateway.agatdemo.com.
Prepare an Ubuntu Machine:
- Provision an Ubuntu machine for the GPT gateway.
- Open the following ports:
  - 22 (SSH)
  - 443 (HTTPS)
  - 1433 (SQL Server)
  - 5432 (PostgreSQL)
Disable ipv6 (there are some ways, look online)
Obtaining the GPT Gateway Version:
- Login to https://gitlab.com/agat-software/python/gatewaychatgpt/-/releases with user itAgat (Lastpass)
- Download the last zip
- Extract and browse gatewaychatgpt-xxx\gatewaychatgpt-xxx\project\config
Configuration Changes:
a. docker.env:

Modify environment variables as needed.

For private AI set LOCAL_AI_API=http://<LLM-ip:port>
For OpenAI set MODEL = gpt-3.5-turbo-16k , For private AI set MODEL=meta-llama/Meta-Llama-3.1-8B-Instruct
If you are using OpenAI - Create API key https://agatsoftware.atlassian.net/wiki/spaces/SKYP/pages/2991423496 and use it in line 25 - OPENAI_API_KEY= sk-W…..
For on-prem environment you can use the general client id and secret in Lastpass “Bgd Gateway general client”
For SaaS environment generate a client ID and secret here: Free Online GUID Generator and document in your BGD Lastpass record
EMBEDDER = (Set openai for GPT or gte-qwen for Local AI)
Database

DBName=CustomerName_Gateway

DBServer=sql_server_container

DBUser=CustomerName_Gateway_User

DBPassword=***

b. gateway.conf:

This is a NGINX configuration file.
Update the server_name property (ports 80 and 443) with your gateway URL.
Example:
server_name bgdXgateway.agatdemo.com www.bgdXgateway.agatdemo.com;

c. postgresql.conf

This is a PostgreSQL configuration file.
Use PGTune to configure PostgreSQL depending on your machine. Read this: https://agatsoftware.atlassian.net/wiki/spaces/VA/pages/3356622865

When using Private AI, move the file compose.override.yml from folder Private AI to folder Gateway.

d. docker-compose-app.service

edit WorkingDirectory=/home/user/Gateway - update with the proper user

e. Edit SQLScripts/FreshInstall/CreateDatabase.sql

Set dbName, dbUserName and dbUserPassword

Move to Gateway server

Upload to the Ubuntu Machine: (Be aware of case sensitivity in Linux)
1. Transfer the gateway folder to the Ubuntu machine's home directory (i.e., ~).
2. Tip: Use the MobaXterm tool's UI for an SSH connection to assist with the transfer.
Installation and Setup on the Ubuntu Machine:
- Connect to the machine and navigate to the home directory:
  cd ~
- Resolving Windows to Unix Copy Issues:
  sudo apt install -y dos2unix sudo dos2unix gateway/InstallScripts/FreshInstall/install.sh sudo dos2unix gateway/gateway.conf
- Install docker - Documentation here (use apt method) [Note that Nvidia deep learning base AMI already included docker]
- Install docker compose plugin - Documentation here (use apt method)
- Login to Docker BusinessGPT account (Password in LP):
  sudo docker login -u businessgpt
- Grant execution permissions and run the installation script (login to docker account using “Docker BusinessGPT” in LP): (case sensitive)
  sudo chmod +x Gateway/InstallScripts/FreshInstall/install.sh sudo ./Gateway/InstallScripts/FreshInstall/install.sh
Installing Gateway database:

cd Gateway
sudo docker compose up mssql -d

Connect to the database with SSMS from office (with ip address of the Gateway)

username: sa, password - see sa password in docker.env

Run CreateDatabase.sql (take from Sqlsripts\freshinstall)

Install service docker-compose-app.service

edit user and run:

sudo cp /home/user/Gateway/docker-compose-app.service /etc/systemd/system/docker-compose-app.service

then run:

sudo systemctl daemon-reload
sudo systemctl start docker-compose-app.service
sudo systemctl enable docker-compose-app.service

Health Check:
- Browse to: {DOMAIN}/v1/alive (replace {DOMAIN} with your hostname).
- Ensure the result displays: “Yes”.

Ensure that the gateway is securely connected by verifying the presence of a green lock icon in the browser

Troubleshooting

If you change anything in docker.env or docker.compose, run this to restart containers

Ubuntu

sudo docker compose up -d

Amazon Linux

sudo docker-compose up -d

Check gateway docker status:

Ubuntu

sudo docker compose ps

Amazon Linux

sudo docker-compose ps

Check gateway logs:

Ubuntu:

sudo docker compose logs --tail=50 -f

Amazon Linux:

sudo docker-compose logs --tail=50 -f

Check system logs:

Ubuntu:

tail -f /var/log/syslog

Amazon Linux:

journalctl -f

Check GPU memory (for Private AI)

watch -n 1 nvidia-smi

If PostgreSQL not configured in docker.env and you already run docker compose - you will need to delete PostgreSQL volume (this will delete all embedding)

Ubuntu:

sudo docker compose down
sudo docker volume rm gateway_postgres-data

Amazon Linux:

sudo docker-compose down
sudo docker volume rm gateway_postgres-data

How to find Gateway version

To locate the version of the Gateway within your Gateway container, navigate to the /gateway/main.py file. You can retrieve the Gateway version by executing the following command:

sudo docker exec -it gateway_api_container python -c "with open('/gateway/version.py', 'r') as f: lines = f.readlines(); print([line.split('\"')[1] for line in lines if '__version__' in line][0])"

Import logs from docker to current location:

First locate location of relevant log

sudo ls /var/lib/docker/volumes/

You might need logs from these locations:

gateway_gateway-data

gateway_doc-loader-data

Copy to current location example:

sudo cp /var/lib/docker/volumes/ubuntu_gateway-data/_data/logs/gatewayQueueDocs/2024-08-01.log .