AWS
Purpose | Option 1 (OSS 20B + GPU embedding) | Option 2 (OSS 20B + CPU embedding) | Option 3 OSS 120B / Gemma 27B + GPU Embedding | Option 4 AWS Bedrock Models |
|---|---|---|---|---|
Gateway (Linux) | g5.xlarge $1.18 | t3a.xlarge. Embedding uses CPU of Gateway. $0.1715 | Co-located with LLM | t3a.xlarge. Embedding uses CPU of Gateway. $0.1715 |
Local LLM (Linux) | g5.xlarge $1.18 | g5.xlarge $1.18 | g5.12xlarge $6.65 | -- |
Dashboard / Ingestor (Windows) | t3a.large $0.2451 | t3a.large $0.2451 | t3a.large $0.2451 | t3a.large $0.2451 |
Total Per Hour | $2.60 | $1.60 | $6.89 | $0.42 |
Per Month 9 hrs x 20 days | $468 | $288 | $1241 | $75 + Bedrock costs |
Option available for solution to start automatically when used, for cost savings. Uses Lambda scripts.
See here for more details specs regarding disk size and other requirements
https://agatsoftware.atlassian.net/wiki/spaces/VA/pages/3125706896
Instances available
GPU options are much more limited - basically only 1 relevant - g5.xlarge with 24GB of VRAM. $1.18 USD /hr excluding VAT.
More expensive than usual gateway GPU ($0.52) and LLM ($0.80)
Will need to deploy lamdas etc to Israel. Route53 and cloudfront are global and not region specific. Only required for start/stop.
Bedrock is not available in the Israel region.