8B

Benchmark Question:
Prompt: 7000 tokens
Completion: 214 tokens

...

* Discounts available for different usage patterns

70B

Option A

Model: neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8

GPU: 4 x RTX 4090 96GB VRAM (84 GB used)

72GB disk space used by model

Tested on Runpod

--model meta-llama/Meta-Llama-3.1-70B-Instruct --gpu-memory-utilization 0.95 --tensor-parallel-size 4--max-model-len 8000

8x 4090 runs full context window at 90% vram. 20k window at 87%. (64GB RAM)

4x 4090 20k window: works (32GB RAM)

2x 4090 20k window: doesn’t work

2x 4090 8k window: doesn’t work

4x 4090 full window: doesn’t work

1x H100 NVL: 20k window: