Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
@@ -0,0 +1,150 @@
|
||||
---
|
||||
name: hermes-vision-backup
|
||||
description: "Backup and restore Hermes auxiliary vision provider configuration so vision_analyze can be recovered after provider changes or profile resets."
|
||||
version: 1.0.0
|
||||
author: ShoNuff
|
||||
platforms: [linux]
|
||||
tags: [hermes, vision, backup, recovery, auxiliary]
|
||||
---
|
||||
|
||||
# Hermes Vision Backup & Recovery
|
||||
|
||||
Hermes uses an **auxiliary vision model** for image analysis when the primary conversation model doesn't support native vision. This is configured via `auxiliary.vision.*` in `config.yaml`. If the vision provider breaks (API key change, provider outage, config reset), `vision_analyze` and `browser_vision` fail.
|
||||
|
||||
## What to Back Up
|
||||
|
||||
The vision configuration lives in two places:
|
||||
|
||||
### 1. Config.yaml — auxiliary.vision section
|
||||
|
||||
```yaml
|
||||
auxiliary:
|
||||
vision:
|
||||
provider: "custom:name" # or openrouter, google, nous, etc.
|
||||
model: "model-name" # e.g. "gpt-4o", "claude-sonnet-4", etc.
|
||||
```
|
||||
|
||||
### 2. The API key for the vision provider
|
||||
|
||||
The vision provider uses the same credential pools as the main provider — but if the vision provider is DIFFERENT from the main provider, its API key must be set separately.
|
||||
|
||||
## Current Setup (as of July 2026)
|
||||
|
||||
**Profile:** default
|
||||
**Main provider:** admin-ai (custom, deepseek-chat via LiteLLM proxy)
|
||||
**Vision provider:** admin-ai (same proxy, different model)
|
||||
**Vision model:** `gemini-flash-latest` (via admin-ai — free tier, fast, vision-capable)
|
||||
**Status:** ✅ Working (fixed: api_key had to be set explicitly — see pitfalls below)
|
||||
|
||||
### Recommended: Gemini Flash via admin-ai (Jul 12, 2026)
|
||||
|
||||
The best current vision setup uses Gemini Flash through the admin-ai LiteLLM proxy. Gemini has a generous free tier (60 req/min) and is already configured in admin-ai:
|
||||
|
||||
```bash
|
||||
hermes config set auxiliary.vision.provider admin-ai
|
||||
hermes config set auxiliary.vision.model gemini-flash-latest
|
||||
```
|
||||
|
||||
No additional API key needed — admin-ai already has the Gemini key. Verify with:
|
||||
```bash
|
||||
curl -s <admin-ai-url>/v1/models | grep -i gemini
|
||||
# Should show: gemini-flash-latest, gemini-pro-latest, etc.
|
||||
```
|
||||
|
||||
**Note:** `vision.provider` (top-level) is NOT the same as `auxiliary.vision.provider`. The `vision_analyze` tool reads from `auxiliary.vision.*`. Setting only the top-level `vision.*` has no effect on image analysis.
|
||||
|
||||
### Verify vision is working
|
||||
|
||||
```python
|
||||
# Test that the configured vision model accepts image inputs:
|
||||
curl -s <admin-ai-url>/v1/chat/completions \
|
||||
-H "Authorization: Bearer <key>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "gemini-flash-latest",
|
||||
"messages": [{"role":"user","content":[
|
||||
{"type":"text","text":"describe this image"},
|
||||
{"type":"image_url","image_url":{"url":"https://example.com/test.png"}}
|
||||
]}]
|
||||
}'
|
||||
```
|
||||
|
||||
Look for a successful response containing `choices[0].message.content` to confirm the model supports vision through the proxy.
|
||||
|
||||
## Recovery Steps
|
||||
|
||||
### Quick fix — set a working vision provider
|
||||
|
||||
```bash
|
||||
# Option A: Use a free/trial vision provider
|
||||
hermes config set auxiliary.vision.provider openrouter
|
||||
hermes config set auxiliary.vision.model openai/gpt-4o
|
||||
|
||||
# Option B: Use Anthropic (requires ANTHROPIC_API_KEY)
|
||||
hermes config set auxiliary.vision.provider anthropic
|
||||
hermes config set auxiliary.vision.model claude-sonnet-4
|
||||
|
||||
# Option C: Use Google Gemini (requires GOOGLE_API_KEY)
|
||||
hermes config set auxiliary.vision.provider google
|
||||
hermes config set auxiliary.vision.model gemini-2.0-flash
|
||||
```
|
||||
|
||||
After setting: `/reset` (new session) for the change to take effect.
|
||||
|
||||
### Custom provider pattern (LiteLLM / Open WebUI proxy)
|
||||
|
||||
When vision is routed through a custom provider (e.g. admin-ai.itpropartner.com running LiteLLM), you MUST set ALL four fields:
|
||||
|
||||
```bash
|
||||
hermes config set auxiliary.vision.provider admin-ai
|
||||
hermes config set auxiliary.vision.base_url https://admin-ai.itpropartner.com/v1
|
||||
hermes config set auxiliary.vision.model openrouter/openai/gpt-4o
|
||||
hermes config set auxiliary.vision.api_key sk-...your-key...
|
||||
```
|
||||
|
||||
The `api_key` field does NOT inherit from the main provider automatically — even if the main provider uses the same proxy. It must be set explicitly on the auxiliary config.
|
||||
|
||||
**Verifying the custom provider has a vision model available:**
|
||||
```bash
|
||||
KEY=<your-api-key>
|
||||
curl -s <base-url>/models -H "Authorization: Bearer $KEY"
|
||||
# Look for models containing: gpt-4o, gpt-4.1, claude-sonnet-4, gemini-2.0-flash, qwen-vl-plus, etc.
|
||||
# Text-only models (deepseek-chat, etc.) will NOT work for vision.
|
||||
```
|
||||
|
||||
### LiteLLM proxy: empty model list
|
||||
|
||||
If `/v1/models` returns an empty `data` array, the API key is wrong or the LiteLLM proxy isn't configured with model routing. Check:
|
||||
1. The API key has access to the `/v1/models` endpoint
|
||||
2. LiteLLM config has models listed in `model_list`
|
||||
3. The vision model is exposed under the expected model ID (e.g. `openrouter/openai/gpt-4o`)
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
hermes config check
|
||||
hermes doctor --fix
|
||||
```
|
||||
|
||||
Then test with a simple image in a new session.
|
||||
|
||||
## Backup Commands
|
||||
|
||||
```bash
|
||||
# Save current vision config
|
||||
hermes config | grep -A4 "auxiliary" > /root/.hermes/.backups/vision-config.txt
|
||||
|
||||
# Include in daily Hermes backup (already covered by hermes-backup.sh)
|
||||
# Vision config is part of config.yaml which is already in the backup tarball
|
||||
```
|
||||
|
||||
The `hermes-backup.sh` script already backs up the entire `~/.hermes/` config directory — so the vision config IS already backed up to Wasabi S3 daily. Recovery is just: restore config.yaml from S3, restart Hermes.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **`auxiliary.vision.provider: auto`** means Hermes tries to auto-detect. If no vision provider is configured, `auto` returns nothing — silent failure.
|
||||
- **Setting a vision provider requires a `/reset` or new session** — mid-conversation changes don't take effect.
|
||||
- **If vision model is incompatible** with the provider (e.g. putting a text-only model in `auxiliary.vision.model`), it fails silently. Test after every change.
|
||||
- **The vision provider can be different from the main provider** — main could be deepseek (no vision) while auxiliary vision is gpt-4o (has vision). This is the expected pattern.
|
||||
- **`auxiliary.vision.api_key` does NOT inherit from the main provider**, even when both use the same provider name. The LiteLLM proxy sends `no-****required` / blank key if `api_key` is left empty on the auxiliary config. Must set it explicitly.
|
||||
- **LiteLLM proxy models** require the full model string as passed from the proxy (e.g. `openrouter/openai/gpt-4o`, not just `gpt-4o`). Check `/v1/models` for the exact ID.
|
||||
Reference in New Issue
Block a user