/
Installing GPT Gateway / LLM with Local AI (public)

Installing GPT Gateway / LLM with Local AI (public)

Translations

Deploying on AWS

  1. Use Deep Learning Base OSS Nvidia Driver GPU AMI.
    See image description to make sure it supports desired instance. e.g. G4dn or G6.
    Take most recent option, usually released in last month. - date at end of image name. E.g. 20240520.

    1. image-20250319-092208.png

      Gateway: Select G4dn.xlarge for Gateway or similar
      LLm - Select G6.xlarge for LLM or similar

    2. Select a key pair you have access to or create a new one

    3. Put in same VPC as other servers for the environment

    4. See below for security group port requirements

    5. 110GB suggested SSD GP3 disk size

  2. Gateway Only

    1. docker login -u businessgpt -p dckr_pat_R1ru2t8mOxClFjkBHjFp_ZGvesc

    2. Open port 8080/443 for Gateway API, 5432 for PostgreSQL (if using multiple gateway servers)

    3. Put docker-compose.yml and compose.override.yml user home directory.

      1. You can use an alternate override file (e.g. compose.override.GatewayInOne.yml), but make sure to rename it to compose.override.yml.

    4. Put gateway-local-llm.env and postgresql.conf in the same directory and configure with relevant settings. If you forget to copy the postgresql.conf file, you’ll need to delete the postgres volume and restart the container.

  3. LLM Only

    1. Open port 5000 for API

    2. Put docker-compose-llm.yml in user home directory. Rename it to docker-compose.yml.

  4. Run “docker compose up -d” and fix any errors.

  5. Create a service that runs docker compose on startup.
    Modify docker-compose-app.service working directory to point to the path where the Gateway docker compose file is located.
    E.g.

    WorkingDirectory=/home/<your-user>/gateway


    Put docker-compose-app.service in user home directory.

sudo cp /home/ubuntu/docker-compose-app.service /etc/systemd/system/docker-compose-app.service
sudo systemctl daemon-reload sudo systemctl start docker-compose-app.service sudo systemctl enable docker-compose-app.service

  1. Validating deployment

nvidia-smi docker stats docker compose logs --tail=50 -f sudo cp /var/lib/docker/volumes/ubuntu_gateway-data/_data/logs/gatewayQueueDocs/Gateway_2024-08-01.log .

 

Remove unused CUDA versions to clear disk space

sudo rm -rf /usr/local/cuda-12.3
sudo rm -rf /usr/local/cuda-12.5
sudo rm -rf /usr/local/cuda-12.6

Remove stopped containers and all unused images

docker system prune -a