hermes-skills/skills/research/web-research-fallback/references/git-raw-docs-extraction.md

Git-Hosted Raw Documentation Extraction

Problem

When researching a project's API or reference documentation, the official rendered HTML site may:

• Require JavaScript (SPA/Next.js) making curl scraping produce only boilerplate
• Have complex HTML layout that's hard to parse
• Be blocked by bot protection (Cloudflare, DataDome)
• Require authentication to view raw markup

Many projects host their documentation source in a Git repository that serves raw Markdown files without JS rendering.

Technique

Step 1: Check if docs are in a public Git repo

Look for links in the page source:

curl -sL "https://docs.example.com" | grep -iE '(git|github|gitlab|bitbucket|raw|markdown|\.md)'

Also check the page footer or "Edit this page" links — they often link directly to the source file.

Step 2: Find the raw file URL pattern

Raw file URLs differ by platform:

Platform	Raw URL pattern
GitHub	https://raw.githubusercontent.com/{owner}/{repo}/{branch}/{path}
GitLab	https://gitlab.com/{owner}/{repo}/-/raw/{branch}/{path}
Bitbucket	https://bitbucket.org/{owner}/{repo}/raw/{branch}/{path}
Zabbix Git (Atlassian Stash/Bitbucket Server)	https://git.zabbix.com/projects/{PROJECT}/repos/{REPO}/raw/{path}?at=refs/heads/{branch}
Self-hosted Gitea/Gogs	https://git.example.com/{owner}/{repo}/raw/branch/{path}

Step 3: Download the raw Markdown

# Generic pattern
curl -sL -o /tmp/docs.md "https://raw.githubusercontent.com/owner/repo/branch/path/to/file.md"

# Example from this session: Zabbix docs on self-hosted Bitbucket
curl -sL -o /tmp/host.md \
  "https://git.zabbix.com/projects/WEB/repos/documentation/raw/en/manual/api/reference/host.md?at=refs/heads/release/7.4"

Step 4: Batch download multiple docs

When you need a whole API reference section:

BASE="https://git.zabbix.com/projects/WEB/repos/documentation/raw/en/manual/api/reference"
BRANCH="refs/heads/release/7.4"
mkdir -p /tmp/zabbix_api

for f in host.md item.md trigger.md history.md event.md; do
    curl -sL -o "/tmp/zabbix_api/$f" "$BASE/$f?at=$BRANCH"
done

Step 5: Determine the branch name

• Current stable: often refs/heads/release/{version} (e.g., release/7.4)
• Next/dev: main, master, develop, or refs/heads/devel
• Check the version selector on the HTML docs page — it will tell you what versions exist
• Try HEAD first if unsure: ?at=refs/heads/main or ?at=refs/heads/master

When This Works Best

• Well-established open-source projects (Zabbix, Nginx, Ansible, PostgreSQL docs, Kubernetes)
• Projects using Sphinx, MkDocs, or similar that render Markdown → HTML — the source is always .md files
• API documentation — method references, object schemas, parameters are usually in clean Markdown tables
• Vendors with self-hosted Git (Atlassian Stash/Bitbucket Server, GitLab CE/EE, Gitea)

When This Does NOT Work

• JS-only documentation (Swagger/OpenAPI generated UIs, ReadMe.io, Postman docs) — there is no Markdown source
• Vendors who only offer HTML docs (rare for good API docs, but happens)
• Proprietary PDF-only documentation
• Wiki platforms (Confluence, Notion, MediaWiki) — these have their own export formats

Pitfalls

• Authentication gates — Some Bitbucket/GitLab instances require login for raw files (as happened in this session with reference_objects.md). The HTML pages loaded fine; the raw Git blob required auth. Try the HTML page first, fall back to Git raw.
• Branch mismatch — The current docs may be on a different branch than main. Check the version selector on the HTML page to find the right ref.
• File reorganization — The Git repo may have a different directory structure than the rendered URL. E.g., Zabbix docs at /en/manual/api/reference/host correspond to Git path /en/manual/api/reference/host.md.
• Relative links — Raw Markdown files contain relative links that won't resolve unless you download the whole tree. For method references, download parent + child files separately.
• Large repos — Don't git clone for docs. Just fetch individual files with curl.

Combined Workflow: HTML Page + Git Raw

Best approach from this session's Zabbix research:

1. Fetch HTML page with curl to find the version/branch selector and verify content exists
2. Extract the Git link from the HTML (Zabbix docs embed <link rel="alternate" type="text/markdown" href="https://git.zabbix.com/...">)
3. Download raw Markdown from the Git blob URL for clean, parseable content
4. Batch download related files for the full API reference section
5. Compile the report from clean Markdown sources rather than HTML-stripped text