Mining Structured API Documentation from SPA Doc Sites

When researching third-party APIs, their docs are often served as single-page applications (apidoc-generated, Slate, ReadMe.io, SwaggerUI). The visible browser page is JS-rendered, but the raw data payload is typically available as static JSON/JS files. This reference documents techniques to find and extract them.

Common SPA Doc Frameworks & Their Data Files

1. apidocjs (http://apidocjs.com)

Framework: Uses Handlebars templates + a JSON data file. The page is an SPA that loads all endpoint data into memory.

Data files to look for:

File	Contents
`api_data.json`	Array of endpoint objects with type, url, title, parameters, responses, permissions
`api_data.js`	Same data wrapped in `define({ "api": [...] })` (RequireJS/AMD)
`api_project.json`	Project metadata: name, version, URL, group ordering
`api_project.js`	Project metadata in AMD wrapper
`api_doc_collection.json`	Postman v2 collection with bodies, headers, URLs, test scripts

Extraction technique:

# Try the JSON version first (cleaner)
curl -sL "https://apidocs.example.com/api_data.json" | python3 -m json.tool

# If 404, try the JS version (has trailing commas — needs cleanup)
curl -sL "https://apidocs.example.com/api_data.js" -o /tmp/data.js

# Strip the AMD wrapper and fix trailing commas
python3 -c "
import re, json
with open('/tmp/data.js') as f:
    content = f.read()
# Extract JSON array from define({ 'api': [...] })
match = re.search(r'define\(\{ \"api\": (\[.*?\])\s*\}\);?\s*$', content, re.DOTALL)
if match:
    raw = match.group(1)
else:
    # Fallback: find outermost array
    start = content.index('[{')
    end = content.rindex('}]') + 2
    raw = content[start:end]
# Fix trailing commas (json doesn't allow them)
raw = re.sub(r',\s*}', '}', raw)
raw = re.sub(r',\s*]', ']', raw)
data = json.loads(raw)
print(json.dumps(data, indent=2)[:5000])
"

What you get: Full endpoint enumeration with:

HTTP method, URL template
Required/optional parameters with types
Response field schemas
Permission/scoping levels
Success/error response patterns
Example request bodies

2. Postman Collections

Many doc sites offer a Postman v2 collection JSON as a downloadable test suite.

Finding it: Check for:

api_doc_collection.json
postman_collection.json
A "Download Postman Collection" link in the docs

Extraction:

import json

with open('/tmp/postman.json') as f:
    data = json.load(f)

def find_all_items(items):
    results = []
    for item in items:
        name = item.get('name', '')
        if 'item' in item:
            results.extend(find_all_items(item['item']))
        else:
            # Leaf node = actual API endpoint
            results.append((name, item))
    return results

items = find_all_items(data.get('item', []))

# Group endpoints by folder
folders = {}
for item in data.get('item', []):
    folder_name = item.get('name', '')
    if 'item' in item:
        for ep in item['item']:
            # Extract URL template
            req = ep.get('request', {})
            url = req.get('url', {})
            if isinstance(url, dict):
                url_template = url.get('raw', 'N/A')
            else:
                url_template = str(url)
            
            # Extract form params
            body = req.get('body', {})
            params = []
            if isinstance(body, dict):
                for mode in ['formdata', 'urlencoded']:
                    for p in body.get(mode, []):
                        params.append(p.get('key'))
            
            folders.setdefault(folder_name, [])
            folders[folder_name].append({
                'name': ep.get('name'),
                'method': req.get('method', 'POST'),
                'url': url_template,
                'params': params
            })

What you get from Postman collections that raw API JSON may not include:

Actual request body templates with placeholder values
Real URL patterns including base URL templates ({{api_url}})
Authorization header templates
Response status codes and example bodies
Pre-request scripts and test assertions

3. Other Doc Frameworks — Quick Reference

Framework	Look for
Swagger/OpenAPI 2.0	`swagger.json`, `api-docs`, `swagger.yaml`
OpenAPI 3.0	`openapi.json`, `openapi.yaml`, `v3/api-docs`
Slate/Middleman	No JSON payload — scrape rendered HTML or look for `source/includes/`
ReadMe.io	Fetch with `?format=json` param, or look for `__NEXT_DATA__` in page HTML
Redoc/RapiDoc	Usually loads an external OpenAPI spec URL — find the `<redoc spec-url="...">` in HTML
Stoplight/Elements	`<elements-api apiDescriptionUrl="...">` in page source

Parsing the Project Metadata

The api_project.json or api_project.js tells you the API's:

Base URL (url field)
API version
Ordered endpoint groups (order array)
Auth mechanism description (in header.content)

# Sample extraction
with open('/tmp/api_project.json') as f:
    project = json.load(f)
    
base_url = project.get('url', 'N/A')
api_name = project.get('name', 'N/A')
version = project.get('version', 'N/A')
groups_in_order = project.get('order', [])

Grouping and Aggregation

Once you have the full endpoint list, aggregate by group to produce a high-level inventory:

=== Agent (6 endpoints)
  POST ?object=agent&action=count   | Count Agents       | Scope: Reseller
  POST ?object=agent&action=create  | Create an Agent    | Scope: Reseller
  POST ?object=agent&action=delete  | Delete Agents      | Scope: Reseller
  POST ?object=agent&action=read    | Read Agents        | Scope: OMP
  POST ?object=agent&action=update  | Update Agents      | Scope: Reseller

=== Domain (6 endpoints)
  POST ?object=domain&action=count   | Count Domains          | Scope: OMP
  POST ?object=domain&action=create  | Create Domain          | Scope: OMP
  POST ?object=domain&action=read    | Read Domain            | Scope: OMP
  POST ?object=domain&action=read&billing=yes | Read Billing | Scope: OMP
  POST ?object=domain&action=update  | Update Domain          | Scope: OMP
  POST ?object=domain&action=delete  | Delete Domain          | Scope: OMP

Auth & Scope Mapping

Most API doc sites define permission/scope levels per endpoint. Extract a scope matrix:

scopes = {}
for entry in data:
    perm_names = [p['name'] for p in entry.get('permission', [])]
    group = entry.get('group', '')
    for perm in perm_names:
        scopes.setdefault(perm, set()).add(group)

for scope, groups in sorted(scopes.items()):
    print(f"{scope}: {', '.join(sorted(groups))}")

This tells you which auth level can access which functional area — critical for integration planning.

Pitfalls

api_data.json with trailing commas: Not valid JSON. The JS AMD wrapper version (api_data.js) often has the same problem. Always fix trailing commas before parsing.
Huge files: Some doc sites have 800K+ JSON files with 30K+ lines. Don't dump the whole thing at once — parse programmatically and extract the structured summary.
URL templates: URLs in the JSON often have template variables like {{api_url}} or use relative paths (?object=x&action=y). The base URL is in api_project.json.
Missing groups: Swagger/OpenAPI may have paths that don't correspond to clean groups — you'll need to aggregate by tag or path prefix.
Authentication: Doc site data files are public and contain endpoint schemas but NOT real credentials. Don't confuse the two.
Version drift: The Postman collection and api_data.json may be out of sync. Cross-reference when possible.

7.7 KiB Raw Permalink Blame History