# Git-Hosted Raw Documentation Extraction

## Problem

When researching a project's API or reference documentation, the official rendered HTML site may:
- Require JavaScript (SPA/Next.js) making curl scraping produce only boilerplate
- Have complex HTML layout that's hard to parse
- Be blocked by bot protection (Cloudflare, DataDome)
- Require authentication to view raw markup

Many projects **host their documentation source in a Git repository** that serves raw Markdown files without JS rendering.

## Technique

### Step 1: Check if docs are in a public Git repo

Look for links in the page source:
```bash
curl -sL "https://docs.example.com" | grep -iE '(git|github|gitlab|bitbucket|raw|markdown|\.md)'
```

Also check the page footer or "Edit this page" links — they often link directly to the source file.

### Step 2: Find the raw file URL pattern

Raw file URLs differ by platform:

| Platform | Raw URL pattern |
|---|---|
| **GitHub** | `https://raw.githubusercontent.com/{owner}/{repo}/{branch}/{path}` |
| **GitLab** | `https://gitlab.com/{owner}/{repo}/-/raw/{branch}/{path}` |
| **Bitbucket** | `https://bitbucket.org/{owner}/{repo}/raw/{branch}/{path}` |
| **Zabbix Git (Atlassian Stash/Bitbucket Server)** | `https://git.zabbix.com/projects/{PROJECT}/repos/{REPO}/raw/{path}?at=refs/heads/{branch}` |
| **Self-hosted Gitea/Gogs** | `https://git.example.com/{owner}/{repo}/raw/branch/{path}` |

### Step 3: Download the raw Markdown

```bash
# Generic pattern
curl -sL -o /tmp/docs.md "https://raw.githubusercontent.com/owner/repo/branch/path/to/file.md"

# Example from this session: Zabbix docs on self-hosted Bitbucket
curl -sL -o /tmp/host.md \
  "https://git.zabbix.com/projects/WEB/repos/documentation/raw/en/manual/api/reference/host.md?at=refs/heads/release/7.4"
```

### Step 4: Batch download multiple docs

When you need a whole API reference section:

```bash
BASE="https://git.zabbix.com/projects/WEB/repos/documentation/raw/en/manual/api/reference"
BRANCH="refs/heads/release/7.4"
mkdir -p /tmp/zabbix_api

for f in host.md item.md trigger.md history.md event.md; do
    curl -sL -o "/tmp/zabbix_api/$f" "$BASE/$f?at=$BRANCH"
done
```

### Step 5: Determine the branch name

- **Current stable:** often `refs/heads/release/{version}` (e.g., `release/7.4`)
- **Next/dev:** `main`, `master`, `develop`, or `refs/heads/devel`
- Check the version selector on the HTML docs page — it will tell you what versions exist
- Try HEAD first if unsure: `?at=refs/heads/main` or `?at=refs/heads/master`

## When This Works Best

- **Well-established open-source projects** (Zabbix, Nginx, Ansible, PostgreSQL docs, Kubernetes)
- **Projects using Sphinx, MkDocs, or similar** that render Markdown → HTML — the source is always `.md` files
- **API documentation** — method references, object schemas, parameters are usually in clean Markdown tables
- **Vendors with self-hosted Git** (Atlassian Stash/Bitbucket Server, GitLab CE/EE, Gitea)

## When This Does NOT Work

- **JS-only documentation** (Swagger/OpenAPI generated UIs, ReadMe.io, Postman docs) — there is no Markdown source
- **Vendors who only offer HTML docs** (rare for good API docs, but happens)
- **Proprietary PDF-only documentation**
- **Wiki platforms** (Confluence, Notion, MediaWiki) — these have their own export formats

## Pitfalls

- **Authentication gates** — Some Bitbucket/GitLab instances require login for raw files (as happened in this session with `reference_objects.md`). The HTML pages loaded fine; the raw Git blob required auth. Try the HTML page first, fall back to Git raw.
- **Branch mismatch** — The `current` docs may be on a different branch than `main`. Check the version selector on the HTML page to find the right ref.
- **File reorganization** — The Git repo may have a different directory structure than the rendered URL. E.g., Zabbix docs at `/en/manual/api/reference/host` correspond to Git path `/en/manual/api/reference/host.md`.
- **Relative links** — Raw Markdown files contain relative links that won't resolve unless you download the whole tree. For method references, download parent + child files separately.
- **Large repos** — Don't `git clone` for docs. Just fetch individual files with curl.

## Combined Workflow: HTML Page + Git Raw

Best approach from this session's Zabbix research:

1. **Fetch HTML page** with curl to find the version/branch selector and verify content exists
2. **Extract the Git link** from the HTML (Zabbix docs embed `<link rel="alternate" type="text/markdown" href="https://git.zabbix.com/...">`)
3. **Download raw Markdown** from the Git blob URL for clean, parseable content
4. **Batch download** related files for the full API reference section
5. **Compile the report** from clean Markdown sources rather than HTML-stripped text