Llama 3.1 70B LatamGPT SFT

01What it can do

04Install

Install Ollama for your OS. Check the model and its quantization before downloading. Start with 4096 tokens of context, then check placement with ollama ps. A command below is not proof that a test was run on your machine.

02Required memory

Approximate GPU VRAM required to run this model, including 4k tokens of context overhead. For a longer context, add ~1 GB per 8k-token increment.

What hardware do you need for Llama 3.1 70B LatamGPT SFT?

To run Llama 3.1 70B LatamGPT SFT locally with Q4 quantization, you need about 41 GB of VRAM. An option to compare: BOSGAME M5 128GB / 2TB (Ryzen AI Max+ 395) — leave some headroom for the system and context; check engine compatibility with the GPU.

03Expected speed

Tokens generated per second in Q4_K_M, 4k context. Beyond 20 t/s, reading is comfortable. Below 10 t/s, that's just for testing.