Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,98 @@
+# Git-Hosted Raw Documentation Extraction
+
+## Problem
+
+When researching a project's API or reference documentation, the official rendered HTML site may:
+- Require JavaScript (SPA/Next.js) making curl scraping produce only boilerplate
+- Have complex HTML layout that's hard to parse
+- Be blocked by bot protection (Cloudflare, DataDome)
+- Require authentication to view raw markup
+
+Many projects **host their documentation source in a Git repository** that serves raw Markdown files without JS rendering.
+
+## Technique
+
+### Step 1: Check if docs are in a public Git repo
+
+Look for links in the page source:
+```bash
+curl -sL "https://docs.example.com" | grep -iE '(git|github|gitlab|bitbucket|raw|markdown|\.md)'
+```
+
+Also check the page footer or "Edit this page" links — they often link directly to the source file.
+
+### Step 2: Find the raw file URL pattern
+
+Raw file URLs differ by platform:
+
+| Platform | Raw URL pattern |
+|---|---|
+| **GitHub** | `https://raw.githubusercontent.com/{owner}/{repo}/{branch}/{path}` |
+| **GitLab** | `https://gitlab.com/{owner}/{repo}/-/raw/{branch}/{path}` |
+| **Bitbucket** | `https://bitbucket.org/{owner}/{repo}/raw/{branch}/{path}` |
+| **Zabbix Git (Atlassian Stash/Bitbucket Server)** | `https://git.zabbix.com/projects/{PROJECT}/repos/{REPO}/raw/{path}?at=refs/heads/{branch}` |
+| **Self-hosted Gitea/Gogs** | `https://git.example.com/{owner}/{repo}/raw/branch/{path}` |
+
+### Step 3: Download the raw Markdown
+
+```bash
+# Generic pattern
+curl -sL -o /tmp/docs.md "https://raw.githubusercontent.com/owner/repo/branch/path/to/file.md"
+
+# Example from this session: Zabbix docs on self-hosted Bitbucket
+curl -sL -o /tmp/host.md \
+  "https://git.zabbix.com/projects/WEB/repos/documentation/raw/en/manual/api/reference/host.md?at=refs/heads/release/7.4"
+```
+
+### Step 4: Batch download multiple docs
+
+When you need a whole API reference section:
+
+```bash
+BASE="https://git.zabbix.com/projects/WEB/repos/documentation/raw/en/manual/api/reference"
+BRANCH="refs/heads/release/7.4"
+mkdir -p /tmp/zabbix_api
+
+for f in host.md item.md trigger.md history.md event.md; do
+    curl -sL -o "/tmp/zabbix_api/$f" "$BASE/$f?at=$BRANCH"
+done
+```
+
+### Step 5: Determine the branch name
+
+- **Current stable:** often `refs/heads/release/{version}` (e.g., `release/7.4`)
+- **Next/dev:** `main`, `master`, `develop`, or `refs/heads/devel`
+- Check the version selector on the HTML docs page — it will tell you what versions exist
+- Try HEAD first if unsure: `?at=refs/heads/main` or `?at=refs/heads/master`
+
+## When This Works Best
+
+- **Well-established open-source projects** (Zabbix, Nginx, Ansible, PostgreSQL docs, Kubernetes)
+- **Projects using Sphinx, MkDocs, or similar** that render Markdown → HTML — the source is always `.md` files
+- **API documentation** — method references, object schemas, parameters are usually in clean Markdown tables
+- **Vendors with self-hosted Git** (Atlassian Stash/Bitbucket Server, GitLab CE/EE, Gitea)
+
+## When This Does NOT Work
+
+- **JS-only documentation** (Swagger/OpenAPI generated UIs, ReadMe.io, Postman docs) — there is no Markdown source
+- **Vendors who only offer HTML docs** (rare for good API docs, but happens)
+- **Proprietary PDF-only documentation**
+- **Wiki platforms** (Confluence, Notion, MediaWiki) — these have their own export formats
+
+## Pitfalls
+
+- **Authentication gates** — Some Bitbucket/GitLab instances require login for raw files (as happened in this session with `reference_objects.md`). The HTML pages loaded fine; the raw Git blob required auth. Try the HTML page first, fall back to Git raw.
+- **Branch mismatch** — The `current` docs may be on a different branch than `main`. Check the version selector on the HTML page to find the right ref.
+- **File reorganization** — The Git repo may have a different directory structure than the rendered URL. E.g., Zabbix docs at `/en/manual/api/reference/host` correspond to Git path `/en/manual/api/reference/host.md`.
+- **Relative links** — Raw Markdown files contain relative links that won't resolve unless you download the whole tree. For method references, download parent + child files separately.
+- **Large repos** — Don't `git clone` for docs. Just fetch individual files with curl.
+
+## Combined Workflow: HTML Page + Git Raw
+
+Best approach from this session's Zabbix research:
+
+1. **Fetch HTML page** with curl to find the version/branch selector and verify content exists
+2. **Extract the Git link** from the HTML (Zabbix docs embed `<link rel="alternate" type="text/markdown" href="https://git.zabbix.com/...">`)
+3. **Download raw Markdown** from the Git blob URL for clean, parseable content
+4. **Batch download** related files for the full API reference section
+5. **Compile the report** from clean Markdown sources rather than HTML-stripped text