| Table of Contents |
|---|
Containers
Purpose | Option 1 (GPU) | Option 2 (CPU) |
|---|---|---|
Gateway (Linux) (Vector DB, SQL DB, Gateway API) | 16 GB RAM (32 GB for larger databases) 4 cores 150GB SSD | 16 GB RAM 4 cores |
Embedder (Linux) | 16 GB RAM 16 GB GPU 4 cores 60 GB SSD | 16 GB RAM 4 cores |
LLM (Linux) | 16 GB RAM 24 GB GPU 4 cores 80 GB SSD | 16 GB RAM 8 cores |
Dashboard/Ingestor (Windows Server- Not a container) | 8 GB RAM 4 cores 80 GB SSD | 8 GB RAM 4 cores |
| Info |
|---|
|
Sizing
Above server hardware requirements are estimated to be sufficient for up to
50 GB of ingested data
300 Licensed users
20 concurrent users asking questions / 75 questions within a 15 min period.
Add more LLM containers with GPUs to support more concurrent users.
AI model servers
The main hardware cost of the BusinessGPT deployment are the AI servers responsible for answering questions. These utilize GPUs.
These servers will also be used to create embeddings for processing and preparing the data for AI.
Server spec:
Linux server Ubuntu 2CPU, 8GB RAM HDD/SSD with a R/W speed of at least 100MB/s.
GPU: CUDA 11.8+, Min 12GB 24GB RAM.
The disk size should be 30% larger than the original content base file size.
Multiple servers with a load balancer may be used for higher performance / High availability.
Graphics Cards
Below are GPU options that are available for purchase for standard servers. The difference between the cards is the number of questions per minute that can be processed.
...
*** Multiple medium range cards are preferable to a lower quantity of more expensive cards.
Dashboard website
Windows Server 2016 Enterprise (or Higher), 4CPU, 8GB RAM, 50G disk space, IIS installed. HDD/SSD with a R/W speed of at least 50MB/s.
Ingestor service
May be co-located with Dashboard server (above)
Windows Server 2016 Enterprise (or Higher), 4CPU, 8GB RAM, 50G disk space, IIS installed. HDD/SSD with a R/W speed of at least 50MB/s.
Dashboard SQL DB
Configured with SQL file storage
Windows Server 2016 Enterprise (or Higher), 4CPU, 8GB RAM, IIS installed. HDD/SSD with a R/W speed of at least 50MB/s.
...
In case files are uploaded manually to the Dashboard, they are saved in the Dashboard db as is (same size)
Gateway server
Linux server Ubuntu 4CPU, 8GB RAM. HDD/SSD with a R/W speed of at least 100MB/s.
The disk size should be 30% larger than the original content file size.
Server can optionally have an NVIDIA GPU with a minimum of 14GB vRAM for embedding the content.
Offline Installation
File / Container sizes
Gateway Image: 7.5 GB
Vector DB Image: 500 MB
Microsoft SQL DB Image (Optional): 1.6 GB
LLM Image with Model: 17 GB
BG Service + Dashboard installation package: 250 MB