151 lines
6.4 KiB
Markdown
151 lines
6.4 KiB
Markdown
---
|
|
name: hermes-vision-backup
|
|
description: "Backup and restore Hermes auxiliary vision provider configuration so vision_analyze can be recovered after provider changes or profile resets."
|
|
version: 1.0.0
|
|
author: ShoNuff
|
|
platforms: [linux]
|
|
tags: [hermes, vision, backup, recovery, auxiliary]
|
|
---
|
|
|
|
# Hermes Vision Backup & Recovery
|
|
|
|
Hermes uses an **auxiliary vision model** for image analysis when the primary conversation model doesn't support native vision. This is configured via `auxiliary.vision.*` in `config.yaml`. If the vision provider breaks (API key change, provider outage, config reset), `vision_analyze` and `browser_vision` fail.
|
|
|
|
## What to Back Up
|
|
|
|
The vision configuration lives in two places:
|
|
|
|
### 1. Config.yaml — auxiliary.vision section
|
|
|
|
```yaml
|
|
auxiliary:
|
|
vision:
|
|
provider: "custom:name" # or openrouter, google, nous, etc.
|
|
model: "model-name" # e.g. "gpt-4o", "claude-sonnet-4", etc.
|
|
```
|
|
|
|
### 2. The API key for the vision provider
|
|
|
|
The vision provider uses the same credential pools as the main provider — but if the vision provider is DIFFERENT from the main provider, its API key must be set separately.
|
|
|
|
## Current Setup (as of July 2026)
|
|
|
|
**Profile:** default
|
|
**Main provider:** admin-ai (custom, deepseek-chat via LiteLLM proxy)
|
|
**Vision provider:** admin-ai (same proxy, different model)
|
|
**Vision model:** `gemini-flash-latest` (via admin-ai — free tier, fast, vision-capable)
|
|
**Status:** ✅ Working (fixed: api_key had to be set explicitly — see pitfalls below)
|
|
|
|
### Recommended: Gemini Flash via admin-ai (Jul 12, 2026)
|
|
|
|
The best current vision setup uses Gemini Flash through the admin-ai LiteLLM proxy. Gemini has a generous free tier (60 req/min) and is already configured in admin-ai:
|
|
|
|
```bash
|
|
hermes config set auxiliary.vision.provider admin-ai
|
|
hermes config set auxiliary.vision.model gemini-flash-latest
|
|
```
|
|
|
|
No additional API key needed — admin-ai already has the Gemini key. Verify with:
|
|
```bash
|
|
curl -s <admin-ai-url>/v1/models | grep -i gemini
|
|
# Should show: gemini-flash-latest, gemini-pro-latest, etc.
|
|
```
|
|
|
|
**Note:** `vision.provider` (top-level) is NOT the same as `auxiliary.vision.provider`. The `vision_analyze` tool reads from `auxiliary.vision.*`. Setting only the top-level `vision.*` has no effect on image analysis.
|
|
|
|
### Verify vision is working
|
|
|
|
```python
|
|
# Test that the configured vision model accepts image inputs:
|
|
curl -s <admin-ai-url>/v1/chat/completions \
|
|
-H "Authorization: Bearer <key>" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "gemini-flash-latest",
|
|
"messages": [{"role":"user","content":[
|
|
{"type":"text","text":"describe this image"},
|
|
{"type":"image_url","image_url":{"url":"https://example.com/test.png"}}
|
|
]}]
|
|
}'
|
|
```
|
|
|
|
Look for a successful response containing `choices[0].message.content` to confirm the model supports vision through the proxy.
|
|
|
|
## Recovery Steps
|
|
|
|
### Quick fix — set a working vision provider
|
|
|
|
```bash
|
|
# Option A: Use a free/trial vision provider
|
|
hermes config set auxiliary.vision.provider openrouter
|
|
hermes config set auxiliary.vision.model openai/gpt-4o
|
|
|
|
# Option B: Use Anthropic (requires ANTHROPIC_API_KEY)
|
|
hermes config set auxiliary.vision.provider anthropic
|
|
hermes config set auxiliary.vision.model claude-sonnet-4
|
|
|
|
# Option C: Use Google Gemini (requires GOOGLE_API_KEY)
|
|
hermes config set auxiliary.vision.provider google
|
|
hermes config set auxiliary.vision.model gemini-2.0-flash
|
|
```
|
|
|
|
After setting: `/reset` (new session) for the change to take effect.
|
|
|
|
### Custom provider pattern (LiteLLM / Open WebUI proxy)
|
|
|
|
When vision is routed through a custom provider (e.g. admin-ai.itpropartner.com running LiteLLM), you MUST set ALL four fields:
|
|
|
|
```bash
|
|
hermes config set auxiliary.vision.provider admin-ai
|
|
hermes config set auxiliary.vision.base_url https://admin-ai.itpropartner.com/v1
|
|
hermes config set auxiliary.vision.model openrouter/openai/gpt-4o
|
|
hermes config set auxiliary.vision.api_key sk-...your-key...
|
|
```
|
|
|
|
The `api_key` field does NOT inherit from the main provider automatically — even if the main provider uses the same proxy. It must be set explicitly on the auxiliary config.
|
|
|
|
**Verifying the custom provider has a vision model available:**
|
|
```bash
|
|
KEY=<your-api-key>
|
|
curl -s <base-url>/models -H "Authorization: Bearer $KEY"
|
|
# Look for models containing: gpt-4o, gpt-4.1, claude-sonnet-4, gemini-2.0-flash, qwen-vl-plus, etc.
|
|
# Text-only models (deepseek-chat, etc.) will NOT work for vision.
|
|
```
|
|
|
|
### LiteLLM proxy: empty model list
|
|
|
|
If `/v1/models` returns an empty `data` array, the API key is wrong or the LiteLLM proxy isn't configured with model routing. Check:
|
|
1. The API key has access to the `/v1/models` endpoint
|
|
2. LiteLLM config has models listed in `model_list`
|
|
3. The vision model is exposed under the expected model ID (e.g. `openrouter/openai/gpt-4o`)
|
|
|
|
### Verify
|
|
|
|
```bash
|
|
hermes config check
|
|
hermes doctor --fix
|
|
```
|
|
|
|
Then test with a simple image in a new session.
|
|
|
|
## Backup Commands
|
|
|
|
```bash
|
|
# Save current vision config
|
|
hermes config | grep -A4 "auxiliary" > /root/.hermes/.backups/vision-config.txt
|
|
|
|
# Include in daily Hermes backup (already covered by hermes-backup.sh)
|
|
# Vision config is part of config.yaml which is already in the backup tarball
|
|
```
|
|
|
|
The `hermes-backup.sh` script already backs up the entire `~/.hermes/` config directory — so the vision config IS already backed up to Wasabi S3 daily. Recovery is just: restore config.yaml from S3, restart Hermes.
|
|
|
|
## Pitfalls
|
|
|
|
- **`auxiliary.vision.provider: auto`** means Hermes tries to auto-detect. If no vision provider is configured, `auto` returns nothing — silent failure.
|
|
- **Setting a vision provider requires a `/reset` or new session** — mid-conversation changes don't take effect.
|
|
- **If vision model is incompatible** with the provider (e.g. putting a text-only model in `auxiliary.vision.model`), it fails silently. Test after every change.
|
|
- **The vision provider can be different from the main provider** — main could be deepseek (no vision) while auxiliary vision is gpt-4o (has vision). This is the expected pattern.
|
|
- **`auxiliary.vision.api_key` does NOT inherit from the main provider**, even when both use the same provider name. The LiteLLM proxy sends `no-****required` / blank key if `api_key` is left empty on the auxiliary config. Must set it explicitly.
|
|
- **LiteLLM proxy models** require the full model string as passed from the proxy (e.g. `openrouter/openai/gpt-4o`, not just `gpt-4o`). Check `/v1/models` for the exact ID.
|