Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,202 @@
+# Mining Structured API Documentation from SPA Doc Sites
+
+When researching third-party APIs, their docs are often served as single-page applications (apidoc-generated, Slate, ReadMe.io, SwaggerUI). The visible browser page is JS-rendered, but the *raw data payload* is typically available as static JSON/JS files. This reference documents techniques to find and extract them.
+
+## Common SPA Doc Frameworks & Their Data Files
+
+### 1. apidocjs (http://apidocjs.com)
+
+**Framework**: Uses Handlebars templates + a JSON data file. The page is an SPA that loads all endpoint data into memory.
+
+**Data files to look for**:
+
+| File | Contents |
+|------|----------|
+| `api_data.json` | Array of endpoint objects with type, url, title, parameters, responses, permissions |
+| `api_data.js` | Same data wrapped in `define({ "api": [...] })` (RequireJS/AMD) |
+| `api_project.json` | Project metadata: name, version, URL, group ordering |
+| `api_project.js` | Project metadata in AMD wrapper |
+| `api_doc_collection.json` | Postman v2 collection with bodies, headers, URLs, test scripts |
+
+**Extraction technique**:
+```bash
+# Try the JSON version first (cleaner)
+curl -sL "https://apidocs.example.com/api_data.json" | python3 -m json.tool
+
+# If 404, try the JS version (has trailing commas — needs cleanup)
+curl -sL "https://apidocs.example.com/api_data.js" -o /tmp/data.js
+
+# Strip the AMD wrapper and fix trailing commas
+python3 -c "
+import re, json
+with open('/tmp/data.js') as f:
+    content = f.read()
+# Extract JSON array from define({ 'api': [...] })
+match = re.search(r'define\(\{ \"api\": (\[.*?\])\s*\}\);?\s*$', content, re.DOTALL)
+if match:
+    raw = match.group(1)
+else:
+    # Fallback: find outermost array
+    start = content.index('[{')
+    end = content.rindex('}]') + 2
+    raw = content[start:end]
+# Fix trailing commas (json doesn't allow them)
+raw = re.sub(r',\s*}', '}', raw)
+raw = re.sub(r',\s*]', ']', raw)
+data = json.loads(raw)
+print(json.dumps(data, indent=2)[:5000])
+"
+```
+
+**What you get**: Full endpoint enumeration with:
+- HTTP method, URL template
+- Required/optional parameters with types
+- Response field schemas
+- Permission/scoping levels
+- Success/error response patterns
+- Example request bodies
+
+### 2. Postman Collections
+
+Many doc sites offer a Postman v2 collection JSON as a downloadable test suite.
+
+**Finding it**: Check for:
+- `api_doc_collection.json`
+- `postman_collection.json`
+- A "Download Postman Collection" link in the docs
+
+**Extraction**:
+```python
+import json
+
+with open('/tmp/postman.json') as f:
+    data = json.load(f)
+
+def find_all_items(items):
+    results = []
+    for item in items:
+        name = item.get('name', '')
+        if 'item' in item:
+            results.extend(find_all_items(item['item']))
+        else:
+            # Leaf node = actual API endpoint
+            results.append((name, item))
+    return results
+
+items = find_all_items(data.get('item', []))
+
+# Group endpoints by folder
+folders = {}
+for item in data.get('item', []):
+    folder_name = item.get('name', '')
+    if 'item' in item:
+        for ep in item['item']:
+            # Extract URL template
+            req = ep.get('request', {})
+            url = req.get('url', {})
+            if isinstance(url, dict):
+                url_template = url.get('raw', 'N/A')
+            else:
+                url_template = str(url)
+            
+            # Extract form params
+            body = req.get('body', {})
+            params = []
+            if isinstance(body, dict):
+                for mode in ['formdata', 'urlencoded']:
+                    for p in body.get(mode, []):
+                        params.append(p.get('key'))
+            
+            folders.setdefault(folder_name, [])
+            folders[folder_name].append({
+                'name': ep.get('name'),
+                'method': req.get('method', 'POST'),
+                'url': url_template,
+                'params': params
+            })
+```
+
+**What you get** from Postman collections that raw API JSON may not include:
+- Actual request body templates with placeholder values
+- Real URL patterns including base URL templates (`{{api_url}}`)
+- Authorization header templates
+- Response status codes and example bodies
+- Pre-request scripts and test assertions
+
+### 3. Other Doc Frameworks — Quick Reference
+
+| Framework | Look for |
+|-----------|----------|
+| **Swagger/OpenAPI 2.0** | `swagger.json`, `api-docs`, `swagger.yaml` |
+| **OpenAPI 3.0** | `openapi.json`, `openapi.yaml`, `v3/api-docs` |
+| **Slate/Middleman** | No JSON payload — scrape rendered HTML or look for `source/includes/` |
+| **ReadMe.io** | Fetch with `?format=json` param, or look for `__NEXT_DATA__` in page HTML |
+| **Redoc/RapiDoc** | Usually loads an external OpenAPI spec URL — find the `<redoc spec-url="...">` in HTML |
+| **Stoplight/Elements** | `<elements-api apiDescriptionUrl="...">` in page source |
+
+## Parsing the Project Metadata
+
+The `api_project.json` or `api_project.js` tells you the API's:
+- Base URL (`url` field)
+- API version
+- Ordered endpoint groups (`order` array)
+- Auth mechanism description (in `header.content`)
+
+```python
+# Sample extraction
+with open('/tmp/api_project.json') as f:
+    project = json.load(f)
+    
+base_url = project.get('url', 'N/A')
+api_name = project.get('name', 'N/A')
+version = project.get('version', 'N/A')
+groups_in_order = project.get('order', [])
+```
+
+## Grouping and Aggregation
+
+Once you have the full endpoint list, aggregate by group to produce a high-level inventory:
+
+```
+=== Agent (6 endpoints)
+  POST ?object=agent&action=count   | Count Agents       | Scope: Reseller
+  POST ?object=agent&action=create  | Create an Agent    | Scope: Reseller
+  POST ?object=agent&action=delete  | Delete Agents      | Scope: Reseller
+  POST ?object=agent&action=read    | Read Agents        | Scope: OMP
+  POST ?object=agent&action=update  | Update Agents      | Scope: Reseller
+
+=== Domain (6 endpoints)
+  POST ?object=domain&action=count   | Count Domains          | Scope: OMP
+  POST ?object=domain&action=create  | Create Domain          | Scope: OMP
+  POST ?object=domain&action=read    | Read Domain            | Scope: OMP
+  POST ?object=domain&action=read&billing=yes | Read Billing | Scope: OMP
+  POST ?object=domain&action=update  | Update Domain          | Scope: OMP
+  POST ?object=domain&action=delete  | Delete Domain          | Scope: OMP
+```
+
+## Auth & Scope Mapping
+
+Most API doc sites define permission/scope levels per endpoint. Extract a scope matrix:
+
+```python
+scopes = {}
+for entry in data:
+    perm_names = [p['name'] for p in entry.get('permission', [])]
+    group = entry.get('group', '')
+    for perm in perm_names:
+        scopes.setdefault(perm, set()).add(group)
+
+for scope, groups in sorted(scopes.items()):
+    print(f"{scope}: {', '.join(sorted(groups))}")
+```
+
+This tells you which auth level can access which functional area — critical for integration planning.
+
+## Pitfalls
+
+- **api_data.json with trailing commas**: Not valid JSON. The JS AMD wrapper version (`api_data.js`) often has the same problem. Always fix trailing commas before parsing.
+- **Huge files**: Some doc sites have 800K+ JSON files with 30K+ lines. Don't dump the whole thing at once — parse programmatically and extract the structured summary.
+- **URL templates**: URLs in the JSON often have template variables like `{{api_url}}` or use relative paths (`?object=x&action=y`). The base URL is in `api_project.json`.
+- **Missing groups**: Swagger/OpenAPI may have paths that don't correspond to clean groups — you'll need to aggregate by tag or path prefix.
+- **Authentication**: Doc site data files are public and contain endpoint schemas but NOT real credentials. Don't confuse the two.
+- **Version drift**: The Postman collection and api_data.json may be out of sync. Cross-reference when possible.