hermes-skills/skills/research/web-research-fallback/references/mining-api-doc-sites.md

# Mining Structured API Documentation from SPA Doc Sites

When researching third-party APIs, their docs are often served as single-page applications (apidoc-generated, Slate, ReadMe.io, SwaggerUI). The visible browser page is JS-rendered, but the *raw data payload* is typically available as static JSON/JS files. This reference documents techniques to find and extract them.

## Common SPA Doc Frameworks & Their Data Files

### 1. apidocjs (http://apidocjs.com)

**Framework**: Uses Handlebars templates + a JSON data file. The page is an SPA that loads all endpoint data into memory.

**Data files to look for**:

| File | Contents |
|------|----------|
| `api_data.json` | Array of endpoint objects with type, url, title, parameters, responses, permissions |
| `api_data.js` | Same data wrapped in `define({ "api": [...] })` (RequireJS/AMD) |
| `api_project.json` | Project metadata: name, version, URL, group ordering |
| `api_project.js` | Project metadata in AMD wrapper |
| `api_doc_collection.json` | Postman v2 collection with bodies, headers, URLs, test scripts |

**Extraction technique**:
```bash
# Try the JSON version first (cleaner)
curl -sL "https://apidocs.example.com/api_data.json" | python3 -m json.tool

# If 404, try the JS version (has trailing commas — needs cleanup)
curl -sL "https://apidocs.example.com/api_data.js" -o /tmp/data.js

# Strip the AMD wrapper and fix trailing commas
python3 -c "
import re, json
with open('/tmp/data.js') as f:
    content = f.read()
# Extract JSON array from define({ 'api': [...] })
match = re.search(r'define\(\{ \"api\": (\[.*?\])\s*\}\);?\s*$', content, re.DOTALL)
if match:
    raw = match.group(1)
else:
    # Fallback: find outermost array
    start = content.index('[{')
    end = content.rindex('}]') + 2
    raw = content[start:end]
# Fix trailing commas (json doesn't allow them)
raw = re.sub(r',\s*}', '}', raw)
raw = re.sub(r',\s*]', ']', raw)
data = json.loads(raw)
print(json.dumps(data, indent=2)[:5000])
"
```

**What you get**: Full endpoint enumeration with:
- HTTP method, URL template
- Required/optional parameters with types
- Response field schemas
- Permission/scoping levels
- Success/error response patterns
- Example request bodies

### 2. Postman Collections

Many doc sites offer a Postman v2 collection JSON as a downloadable test suite.

**Finding it**: Check for:
- `api_doc_collection.json`
- `postman_collection.json`
- A "Download Postman Collection" link in the docs

**Extraction**:
```python
import json

with open('/tmp/postman.json') as f:
    data = json.load(f)

def find_all_items(items):
    results = []
    for item in items:
        name = item.get('name', '')
        if 'item' in item:
            results.extend(find_all_items(item['item']))
        else:
            # Leaf node = actual API endpoint
            results.append((name, item))
    return results

items = find_all_items(data.get('item', []))

# Group endpoints by folder
folders = {}
for item in data.get('item', []):
    folder_name = item.get('name', '')
    if 'item' in item:
        for ep in item['item']:
            # Extract URL template
            req = ep.get('request', {})
            url = req.get('url', {})
            if isinstance(url, dict):
                url_template = url.get('raw', 'N/A')
            else:
                url_template = str(url)
            
            # Extract form params
            body = req.get('body', {})
            params = []
            if isinstance(body, dict):
                for mode in ['formdata', 'urlencoded']:
                    for p in body.get(mode, []):
                        params.append(p.get('key'))
            
            folders.setdefault(folder_name, [])
            folders[folder_name].append({
                'name': ep.get('name'),
                'method': req.get('method', 'POST'),
                'url': url_template,
                'params': params
            })
```

**What you get** from Postman collections that raw API JSON may not include:
- Actual request body templates with placeholder values
- Real URL patterns including base URL templates (`{{api_url}}`)
- Authorization header templates
- Response status codes and example bodies
- Pre-request scripts and test assertions

### 3. Other Doc Frameworks — Quick Reference

| Framework | Look for |
|-----------|----------|
| **Swagger/OpenAPI 2.0** | `swagger.json`, `api-docs`, `swagger.yaml` |
| **OpenAPI 3.0** | `openapi.json`, `openapi.yaml`, `v3/api-docs` |
| **Slate/Middleman** | No JSON payload — scrape rendered HTML or look for `source/includes/` |
| **ReadMe.io** | Fetch with `?format=json` param, or look for `__NEXT_DATA__` in page HTML |
| **Redoc/RapiDoc** | Usually loads an external OpenAPI spec URL — find the `<redoc spec-url="...">` in HTML |
| **Stoplight/Elements** | `<elements-api apiDescriptionUrl="...">` in page source |

## Parsing the Project Metadata

The `api_project.json` or `api_project.js` tells you the API's:
- Base URL (`url` field)
- API version
- Ordered endpoint groups (`order` array)
- Auth mechanism description (in `header.content`)

```python
# Sample extraction
with open('/tmp/api_project.json') as f:
    project = json.load(f)
    
base_url = project.get('url', 'N/A')
api_name = project.get('name', 'N/A')
version = project.get('version', 'N/A')
groups_in_order = project.get('order', [])
```

## Grouping and Aggregation

Once you have the full endpoint list, aggregate by group to produce a high-level inventory:

```
=== Agent (6 endpoints)
  POST ?object=agent&action=count   | Count Agents       | Scope: Reseller
  POST ?object=agent&action=create  | Create an Agent    | Scope: Reseller
  POST ?object=agent&action=delete  | Delete Agents      | Scope: Reseller
  POST ?object=agent&action=read    | Read Agents        | Scope: OMP
  POST ?object=agent&action=update  | Update Agents      | Scope: Reseller

=== Domain (6 endpoints)
  POST ?object=domain&action=count   | Count Domains          | Scope: OMP
  POST ?object=domain&action=create  | Create Domain          | Scope: OMP
  POST ?object=domain&action=read    | Read Domain            | Scope: OMP
  POST ?object=domain&action=read&billing=yes | Read Billing | Scope: OMP
  POST ?object=domain&action=update  | Update Domain          | Scope: OMP
  POST ?object=domain&action=delete  | Delete Domain          | Scope: OMP
```

## Auth & Scope Mapping

Most API doc sites define permission/scope levels per endpoint. Extract a scope matrix:

```python
scopes = {}
for entry in data:
    perm_names = [p['name'] for p in entry.get('permission', [])]
    group = entry.get('group', '')
    for perm in perm_names:
        scopes.setdefault(perm, set()).add(group)

for scope, groups in sorted(scopes.items()):
    print(f"{scope}: {', '.join(sorted(groups))}")
```

This tells you which auth level can access which functional area — critical for integration planning.

## Pitfalls

- **api_data.json with trailing commas**: Not valid JSON. The JS AMD wrapper version (`api_data.js`) often has the same problem. Always fix trailing commas before parsing.
- **Huge files**: Some doc sites have 800K+ JSON files with 30K+ lines. Don't dump the whole thing at once — parse programmatically and extract the structured summary.
- **URL templates**: URLs in the JSON often have template variables like `{{api_url}}` or use relative paths (`?object=x&action=y`). The base URL is in `api_project.json`.
- **Missing groups**: Swagger/OpenAPI may have paths that don't correspond to clean groups — you'll need to aggregate by tag or path prefix.
- **Authentication**: Doc site data files are public and contain endpoint schemas but NOT real credentials. Don't confuse the two.
- **Version drift**: The Postman collection and api_data.json may be out of sync. Cross-reference when possible.