36 lines
1.4 KiB
Markdown
36 lines
1.4 KiB
Markdown
# Ollama Model RAM Sizing — Core Server
|
|
|
|
## Server Specs
|
|
- **Model:** netcup RS 2000 G12
|
|
- **RAM:** 15 GB total
|
|
- **vCPUs:** 8 (EPYC 9645)
|
|
- **Swap:** /dev/vzswap (compressed swap — not real RAM)
|
|
|
|
## Model Testing Results (Jul 11, 2026)
|
|
|
|
| Model | Params | Quant | Size on Disk | Loads? | RAM Needed | Notes |
|
|
|-------|--------|-------|-------------|--------|------------|-------|
|
|
| `llama3.2:3b` | 3B | Q4_K_M | 2.0 GB | ✅ Yes | ~3-4 GB | Default local fallback |
|
|
| `llama3.3:70b` | 70B | Q4_K_M | 42 GB | ❌ No | ~35-40 GB | Error: `insufficient memory (attempted to allocate 31320 MB)` |
|
|
|
|
## Diagnostic Commands
|
|
|
|
```bash
|
|
# Show currently LOADED models (models actually in memory)
|
|
ollama ps
|
|
|
|
# Show DOWNLOADED models (models available but may not load)
|
|
ollama list
|
|
|
|
# Check memory usage
|
|
free -h
|
|
|
|
# Check Ollama errors
|
|
journalctl -u ollama --no-pager | grep -i "insufficient memory\|failed to load"
|
|
```
|
|
|
|
## Key Rule
|
|
|
|
`ollama list` shows what's downloaded. `ollama ps` shows what's actually loadable. A model can pass `ollama list` and still be unusable if RAM is insufficient. Always check `ollama ps` + `journalctl -u ollama` before declaring a model operational.
|
|
|
|
For this server (15 GB RAM), the maximum viable model is ~7B params at Q4 quantization (~4-5 GB RAM). 13B models at Q4 may fit (~8 GB) but leave little room for Ollama's overhead + system processes. 70B models are impossible without additional RAM. |