How to install GPT Gateway

By David Elkarat
4 min

GPT Gateway Fresh Install Guide

 

If you’re deploying private AI please refer to this wiki https://agatsoftware.atlassian.net/wiki/spaces/SKYP/pages/3200122882

Use the same image of LLM for the Gateway

If you already have existing environment, please use this tutorial: https://agatsoftware.atlassian.net/wiki/spaces/SKYP/pages/3098050611

 

  1. Setting Up a Hostname:

    • Create a hostname for the GPT gateway API.

    • Example: https://bgdXgateway.agatdemo.com.

  2. Prepare an Ubuntu Machine:

    • Provision an Ubuntu machine for the GPT gateway.

    • Open the following ports:

      • 22 (SSH)

      • 443 (HTTPS)

      • 1433 (SQL Server)

      • 5432 (PostgreSQL)

  3. Disable ipv6 (there are some ways, look online)

  4. Obtaining the GPT Gateway Version:

    • Login to https://gitlab.com/agat-software/python/gatewaychatgpt/-/releases with user itAgat (Lastpass)

    • Download the last zip

      image-20250213-133855.png

    • Extract and browse gatewaychatgpt-xxx\gatewaychatgpt-xxx\project\config

  5. Configuration Changes:

    a. docker.env:

Modify environment variables as needed.

  • For private AI set LOCAL_AI_API=http://<LLM-ip:port>

  • For OpenAI set MODEL = gpt-3.5-turbo-16k , For private AI set MODEL=meta-llama/Meta-Llama-3.1-8B-Instruct

  • If you are using OpenAI - Create API key https://agatsoftware.atlassian.net/wiki/spaces/SKYP/pages/2991423496 and use it in line 25 - OPENAI_API_KEY= sk-W…..

  • For on-prem environment you can use the general client id and secret in Lastpass “Bgd Gateway general client”

  • For SaaS environment generate a client ID and secret here: Free Online GUID Generator and document in your BGD Lastpass record

  • EMBEDDER = (Set openai for GPT or gte-qwen for Local AI)

  • Database

DBName=CustomerName_Gateway

DBServer=sql_server_container

DBUser=CustomerName_Gateway_User

DBPassword=***

b. gateway.conf:

  1. This is a NGINX configuration file.

  2. Update the server_name property (ports 80 and 443) with your gateway URL.

  3. Example:

    server_name bgdXgateway.agatdemo.com www.bgdXgateway.agatdemo.com;

c. postgresql.conf

When using Private AI, move the file compose.override.yml from folder Private AI to folder Gateway.

d. docker-compose-app.service

  • edit WorkingDirectory=/home/user/Gateway - update with the proper user

e. Edit SQLScripts/FreshInstall/CreateDatabase.sql

  • Set dbName, dbUserName and dbUserPassword

Move to Gateway server

  1. Upload to the Ubuntu Machine: (Be aware of case sensitivity in Linux)

    1. Transfer the gateway folder to the Ubuntu machine's home directory (i.e., ~).

    2. Tip: Use the MobaXterm tool's UI for an SSH connection to assist with the transfer.

  2. Installation and Setup on the Ubuntu Machine:

    • Connect to the machine and navigate to the home directory:

      cd ~

    • Resolving Windows to Unix Copy Issues:

      sudo apt install -y dos2unix sudo dos2unix gateway/InstallScripts/FreshInstall/install.sh sudo dos2unix gateway/gateway.conf

    • Install docker - Documentation here (use apt method) [Note that Nvidia deep learning base AMI already included docker]

    • Install docker compose plugin - Documentation here (use apt method)

    • Login to Docker BusinessGPT account (Password in LP):

      sudo docker login -u businessgpt

    • Grant execution permissions and run the installation script (login to docker account using “Docker BusinessGPT” in LP): (case sensitive)

      sudo chmod +x Gateway/InstallScripts/FreshInstall/install.sh sudo ./Gateway/InstallScripts/FreshInstall/install.sh

  3. Installing Gateway database:

cd Gateway sudo docker compose up mssql -d

Connect to the database with SSMS from office (with ip address of the Gateway)

username: sa, password - see sa password in docker.env

Run CreateDatabase.sql (take from Sqlsripts\freshinstall)

 

  1. Install service docker-compose-app.service

edit user and run:

sudo cp /home/user/Gateway/docker-compose-app.service /etc/systemd/system/docker-compose-app.service

then run:

sudo systemctl daemon-reload sudo systemctl start docker-compose-app.service sudo systemctl enable docker-compose-app.service

  1. Health Check:

    • Browse to: {DOMAIN}/v1/alive (replace {DOMAIN} with your hostname).

    • Ensure the result displays: “Yes”.

Ensure that the gateway is securely connected by verifying the presence of a green lock icon in the browser

 

Troubleshooting

If you change anything in docker.env or docker.compose, run this to restart containers

Ubuntu

sudo docker compose up -d

Amazon Linux

sudo docker-compose up -d

 

Check gateway docker status:

Ubuntu

sudo docker compose ps

Amazon Linux

sudo docker-compose ps

 

Check gateway logs:

Ubuntu:

sudo docker compose logs --tail=50 -f

Amazon Linux:

sudo docker-compose logs --tail=50 -f

 

Check system logs:

Ubuntu:

tail -f /var/log/syslog

Amazon Linux:

journalctl -f

 

Check GPU memory (for Private AI)

watch -n 1 nvidia-smi

 

If PostgreSQL not configured in docker.env and you already run docker compose - you will need to delete PostgreSQL volume (this will delete all embedding)

Ubuntu:

sudo docker compose down sudo docker volume rm gateway_postgres-data

Amazon Linux:

sudo docker-compose down sudo docker volume rm gateway_postgres-data

 

How to find Gateway version

To locate the version of the Gateway within your Gateway container, navigate to the /gateway/main.py file. You can retrieve the Gateway version by executing the following command:

sudo docker exec -it gateway_api_container python -c "with open('/gateway/version.py', 'r') as f: lines = f.readlines(); print([line.split('\"')[1] for line in lines if '__version__' in line][0])"

 

Import logs from docker to current location:

First locate location of relevant log

sudo ls /var/lib/docker/volumes/
image-20250119-094459.png

You might need logs from these locations:

gateway_gateway-data

gateway_doc-loader-data

 

Copy to current location example:

sudo cp /var/lib/docker/volumes/ubuntu_gateway-data/_data/logs/gatewayQueueDocs/2024-08-01.log .

Related content