feat: add TouchDesigner integration skill

New skill: creative/touchdesigner — control a running TouchDesigner instance via REST API. Build real-time visual networks programmatically. Architecture: Hermes Agent -> HTTP REST (curl) -> TD WebServer DAT -> TD Python env Key features: - Custom API handler (scripts/custom_api_handler.py) that creates a self-contained WebServer DAT + callback in TD. More reliable than the official mcp_webserver_base.tox which frequently fails module imports. - Discovery-first workflow: never hardcode TD parameter names. Always probe the running instance first since names change across versions. - Persistent setup: save the TD project once with the API handler baked in. TD auto-opens the last project on launch, so port 9981 is live with zero manual steps after first-time setup. - Works via curl in execute_code (no MCP dependency required). - Optional MCP server config for touchdesigner-mcp-server npm package. Skill structure (2823 lines total): SKILL.md (209 lines) — setup, workflow, key rules, operator reference references/pitfalls.md (276 lines) — 24 hard-won lessons references/operators.md (239 lines) — all 6 operator families references/network-patterns.md (589 lines) — audio-reactive, generative, video processing, GLSL, instancing, live performance recipes references/mcp-tools.md (501 lines) — 13 MCP tool schemas references/python-api.md (443 lines) — TD Python scripting patterns references/troubleshooting.md (274 lines) — connection diagnostics scripts/custom_api_handler.py (140 lines) — REST API handler for TD scripts/setup.sh (152 lines) — prerequisite checker Tested on TouchDesigner 099 Non-Commercial (macOS/darwin).
2026-04-27 01:11:40 +00:00 · 2026-04-15 10:33:15 +05:30 · 2026-04-15 10:33:15 +05:30 · 7a5371b20d
commit 7a5371b20d
parent c49a58a6d0
9 changed files with 3277 additions and 0 deletions
--- a/skills/creative/touchdesigner/references/pitfalls.md
+++ b/skills/creative/touchdesigner/references/pitfalls.md
@ -0,0 +1,336 @@
+# TouchDesigner MCP — Pitfalls & Lessons Learned
+
+Hard-won knowledge from real TD sessions. Read this before building anything.
+
+## Setup & Connection
+
+### 1. The .tox from the git repo is BROKEN
+
+The `td/mcp_webserver_base.tox` in the `8beeeaaat/touchdesigner-mcp` git clone is **incomplete**. It's missing the `td_server` Python module (generated by `npm run gen:webserver` which requires Docker). Port 9981 opens, but every route returns 404.
+
+**Always download the release zip:**
+```bash
+curl -L -o td.zip \
+  "https://github.com/8beeeaaat/touchdesigner-mcp/releases/latest/download/touchdesigner-mcp-td.zip"
+unzip -o td.zip -d touchdesigner-mcp-td
+```
+
+### 2. The release .tox also breaks (frequently)
+
+Even the correct release .tox fails after drag-and-drop import because `import_modules.py` resolves `modules/` via `parent().par.externaltox.eval()` — a relative path that often goes wrong. Symptoms: port 9981 listens, all routes 404, TD Textport shows `[ERROR] Failed to setup modules`.
+
+**The custom API handler (`scripts/custom_api_handler.py`) is more reliable.** It has zero external module dependencies — just a WebServer DAT + Text DAT callback. The skill's setup workflow should try the .tox first, test with `curl`, and auto-deploy the handler if 404.
+
+### 3. You CANNOT automate the .tox import from outside TD
+
+TD has no CLI flag to import a .tox. macOS blocks keystroke injection via System Events for security. The only way to get code into TD from outside is:
+- Have a WebServer DAT already running (chicken-and-egg)
+- AppleScript to open Textport + clipboard paste (fragile, not always reliable)
+- User manually drags the .tox or pastes a script
+
+**Plan for one manual step** from the user (either .tox drag-drop or Textport paste). Make it as frictionless as possible: `open -R /path/to/file` to reveal in Finder.
+
+### 4. The npm package name is `touchdesigner-mcp-server` (not `@anthropic/...`)
+
+The Hermes config should use:
+```yaml
+command: npx
+args: ["-y", "touchdesigner-mcp-server@latest"]
+```
+
+### 5. MCP tools may register but not be callable
+
+Hermes may report "17 MCP tool(s) now available" but the tools aren't exposed as function calls. Use the REST API directly via `curl` in `execute_code` as a reliable fallback:
+```python
+def td_exec(script):
+    escaped = json.dumps({"script": script})
+    cmd = f"curl -s -X POST -H 'Content-Type: application/json' -d {shlex.quote(escaped)} 'http://127.0.0.1:9981/api/td/server/exec'"
+    return json.loads(terminal(cmd)['output'])
+```
+
+## TD WebServer DAT Quirks
+
+### 6. Response body goes in `response['data']`, NOT `response['body']`
+
+When writing custom WebServer DAT handlers, the response payload must be set on the `data` key:
+```python
+response['data'] = json.dumps({"result": 42})  # ✓ works
+response['body'] = json.dumps({"result": 42})   # ✗ ignored
+```
+
+### 7. Request POST body comes as BYTES in `request['data']`
+
+Not `request['body']`, and it's `bytes` not `str`:
+```python
+raw = request.get('data', b'')
+if isinstance(raw, bytes):
+    raw = raw.decode('utf-8')
+body = json.loads(raw) if raw else {}
+```
+
+### 8. Non-Commercial license caps resolution at 1280×1280
+
+Setting `resolutionw=1920` silently clamps to 1280. Always check effective resolution after creation:
+```python
+n.cook(force=True)
+actual = str(n.width) + 'x' + str(n.height)
+```
+
+## Parameter Names
+
+### 9. NEVER hardcode parameter names — always discover
+
+Parameter names change between TD versions. What works in 099 may not work in 098 or 2023.x. Always run discovery first:
+```python
+n = root.create(glslTOP, '_test')
+pars = [(p.name, type(p.val).__name__) for p in n.pars()]
+n.destroy()
+```
+
+Known differences from docs/online references:
+| What docs say | TD 099 actual | Notes |
+|---------------|---------------|-------|
+| `dat` | `pixeldat` | GLSL TOP pixel shader DAT |
+| `colora` | `alpha` | Constant TOP alpha |
+| `sizex` / `sizey` | `size` | Blur TOP (single value) |
+| `fontr/g/b/a` | `fontcolorr/g/b/a` | Text TOP font color (r/g/b) |
+| `fontcolora` | `fontalpha` | Text TOP font alpha (NOT `fontcolora`) |
+| `bgcolora` | `bgalpha` | Text TOP bg alpha |
+| `value1name` | `vec0name` | GLSL TOP uniform name |
+
+### 10. Use `safe_par()` pattern for cross-version compatibility
+
+```python
+def safe_par(node, name, value):
+    p = getattr(node.par, name, None)
+    if p is not None:
+        p.val = value
+        return True
+    return False
+```
+
+### 11. `td.tdAttributeError` crashes the whole script
+
+If you do `node.par.nonexistent = value`, TD raises `tdAttributeError` and **stops the entire script**. There's no way to catch it with try/except in some TD versions. Always check with `getattr` first or use `safe_par()`.
+
+## GLSL Shaders
+
+### 12. `uTDCurrentTime` does NOT exist in TD 099
+
+The GLSL builtin for time was removed or never existed in some builds. Feed time via a 1×1 Constant TOP input. **CRITICAL: set format to `rgba32float`** — the default 8-bit format clamps values to 0-1, so `absTime.seconds % 1000.0` gets clamped and the GLSL shader sees a frozen time value of 1.0:
+```python
+t = root.create(constantTOP, 'time_driver')
+t.par.format = 'rgba32float'  # ← REQUIRED! Without this, time is stuck at 1.0
+t.par.outputresolution = 'custom'
+t.par.resolutionw = 1
+t.par.resolutionh = 1
+t.par.colorr.expr = "absTime.seconds % 1000.0"
+t.par.colorg.expr = "int(absTime.seconds / 1000.0)"
+t.outputConnectors[0].connect(glsl.inputConnectors[0])
+```
+In GLSL:
+```glsl
+vec4 td = texture(sTD2DInputs[0], vec2(0.5));
+float t = td.r + td.g * 1000.0;
+```
+
+### 13. GLSL compile errors are silent in the API
+
+The GLSL TOP shows a yellow warning triangle in the UI but `node.errors()` may return empty string. Check `node.warnings()` too, and create an Info DAT pointed at the GLSL TOP to read the actual compiler output.
+
+### 14. TD GLSL uses `vUV.st` not `gl_FragCoord`
+
+Standard GLSL patterns don't work. TD provides:
+- `vUV.st` — UV coordinates (0-1)
+- `uTDOutputInfo.res.zw` — resolution
+- `sTD2DInputs[0]` — input textures
+- `layout(location = 0) out vec4 fragColor` — output
+
+## Node Management
+
+### 15. Destroying nodes while iterating `root.children` causes `tdError`
+
+The iterator is invalidated when a child is destroyed. Always snapshot first:
+```python
+kids = list(root.children)  # snapshot
+for child in kids:
+    if child.valid:  # check — earlier destroys may cascade
+        child.destroy()
+```
+
+### 16. Feedback TOP: use `top` parameter, NOT direct input wire
+
+In TD 099, the feedbackTOP's `top` parameter references which TOP to delay. **Do not also wire that TOP directly into the feedback's input** — this creates a real cook dependency loop (warning flood, potential crash). The "Not enough sources" error on feedbackTOP is benign and resolves after a few frames of playback.
+
+Correct setup:
+```python
+fb = root.create(feedbackTOP, 'fb_delay')
+fb.par.top = comp.path          # reference only — no wire to fb input
+fb.outputConnectors[0].connect(xf)  # fb output -> transform -> fade -> comp
+```
+
+The resulting "Cook dependency loop detected" **warning** on the transform/fade chain is expected and correct — that's what feedback loops do. It's informational, not an error.
+
+### 16. GLSL TOP auto-creates companion nodes
+
+Creating a `glslTOP` also creates `name_pixel` (Text DAT), `name_info` (Info DAT), and `name_compute` (Text DAT). These are visible in the network and count toward node totals. Don't be alarmed by "extra" nodes.
+
+### 17. The default project root is `/project1`
+
+New TD files start with `/project1` as the main container. System nodes live at `/`, `/ui`, `/sys`, `/local`, `/perform`. Don't create user nodes outside `/project1`.
+
+### 18. `open -R` reveals the file but can't automate the drag
+
+Use `open -R /path/to/file.tox` to open Finder highlighting the file. The user must then drag it into TD manually. No AppleScript workaround exists on modern macOS due to accessibility restrictions.
+
+## Workflow
+
+### 19. Always verify after building — errors are silent
+
+Node errors and broken connections produce no output. Always check:
+```python
+for c in list(root.children):
+    e = c.errors()
+    w = c.warnings()
+    if e: print(c.name, 'ERR:', e)
+    if w: print(c.name, 'WARN:', w)
+```
+
+### 20. Build in one big `execute_python_script` call, not many small ones
+
+Each API round-trip adds latency. Bundle node creation + parameter setting + wiring into a single script that creates everything at once, then verify in one final call.
+
+### 21. Window COMP param for display target is `winop` (not `top` or `window`)
+
+To display output in a separate window:
+```python
+win = root.create(windowCOMP, 'display')
+win.par.winop = '/project1/logo_out'  # ← this is the correct param
+win.par.winw = 1280; win.par.winh = 720
+win.par.winopen.pulse()  # open the window
+```
+
+### 22. Save the project to make API persistent across TD restarts
+
+After deploying the custom API handler, save the project:
+```python
+td_exec("project.save(os.path.expanduser('~/Documents/HermesAgent.toe'))")
+```
+TD auto-opens the last saved project on launch. The API handler is now baked into the .toe file — next time TD opens, port 9981 is live with zero manual steps. To explicitly launch with this project: `open /Applications/TouchDesigner.app ~/Documents/HermesAgent.toe`
+
+### 23. `sample()` returns frozen pixels when called from WebServer DAT callback
+
+`out.sample(x, y)` called from inside the API handler's `exec()` returns pixels from a single cook snapshot. It does NOT update between multiple API calls in quick succession. To verify animation is working, either:
+- Compare samples with a 2+ second delay between separate `td_exec()` calls
+- Use `screencapture` on the display window
+- Check `absTime.seconds` is advancing and shader uses time correctly
+
+### 22. `outputresolution` is a string menu, not an integer
+
+### 25. MovieFileOut TOP: H.264/H.265 requires Commercial license
+
+In Non-Commercial TD 099, encoding with H.264 or H.265 produces an error: "GPU Accelerated H.264/H.265 Encoding requires a Commercial license". Use Motion JPEG instead:
+```python
+rec = root.create(moviefileoutTOP, 'recorder')
+rec.par.type = 'movie'
+rec.par.file = '/tmp/output.mov'
+rec.par.videocodec = 'mjpa'  # Motion JPEG — works on Non-Commercial
+```
+
+For image sequences, use `type = 'imagesequence'` and the file param **must** use `me.fileSuffix`:
+```python
+rec.par.type = 'imagesequence'
+rec.par.imagefiletype = 'png'
+rec.par.file.expr = "'/tmp/frames/out' + me.fileSuffix"
+```
+
+### 26. MovieFileOut `.record()` method may not exist
+
+In TD 099, there is no `.record()` method on moviefileoutTOP. Use the toggle parameter instead:
+```python
+rec.par.record = True   # start recording
+rec.par.record = False  # stop recording
+```
+
+When setting the file path and starting recording in the same script, use `run()` with `delayFrames` to avoid a race condition where the old filename is used:
+```python
+rec.par.file = '/tmp/new_output.mov'
+run("op('/project1/recorder').par.record = True", delayFrames=2)
+```
+
+### 27. TOP.save() captures same frame when called rapidly
+
+`op('null1').save(path)` captures the current GPU texture at call time. When called multiple times in a single script (or rapid API calls), TD doesn't cook new frames between saves — all exported PNGs will be identical. To get unique frames, use the MovieFileOut TOP which records in real-time from TD's cook cycle.
+
+### 28. AudioFileIn CHOP: cue before recording for sync
+
+When recording audio-reactive visuals, always cue the audio to the start before beginning the recording. Otherwise the visuals are synced to wherever the audio happens to be in its playback:
+```python
+op('/project1/audio_in').par.cue.pulse()  # reset to start
+run("op('/project1/recorder').par.record = True", delayFrames=3)
+```
+The audio plays via `playmode=0` (Locked to Timeline), so it stays in sync with TD's frame clock. Use `audiodeviceoutCHOP` to hear the audio during recording.
+
+### 29. Audio Spectrum CHOP output is weak — boost with Math CHOP
+
+The raw AudioSpectrum CHOP output has very small values (often 0.001-0.05 range). When fed directly to CHOP To TOP → GLSL, the shader barely reacts. Always insert a Math CHOP with `gain=5` (or higher) between the spectrum and the CHOP To TOP to get usable 0-1 range values in the shader.
+
+### 30. CHOP To TOP texture size — Resample to 256 first
+
+`choptoTOP` creates a texture where width = number of samples. An AudioSpectrum CHOP at 44100Hz has ~24000 samples — creating a 24000×1 texture is wasteful. Use a Resample CHOP set to 256 or 512 samples before the CHOP To TOP for an efficient spectrum texture.
+
+### 31. CHOP To TOP has NO input connectors — use par.chop reference
+
+`choptoTOP` does NOT have input connectors. `resample.outputConnectors[0].connect(chop_to_top.inputConnectors[0])` silently does nothing. Use the `chop` parameter instead:
+```python
+spec_tex = root.create(choptoTOP, 'spectrum_tex')
+spec_tex.par.chop = resample  # ← correct: parameter reference
+# NOT: resample.outputConnectors[0].connect(spec_tex.inputConnectors[0])  # ← WRONG: no connectors
+```
+
+### 22. `outputresolution` is a string menu, not an integer
+
+The `outputresolution` param is a menu with string values:
+```
+menuNames: ['useinput','eighth','quarter','half','2x','4x','8x','fit','limit','custom','parpanel']
+```
+Always use the string form. Setting `outputresolution = 9` may silently fail.
+```python
+node.par.outputresolution = 'custom'  # ✓ correct
+node.par.resolutionw = 1280; node.par.resolutionh = 720
+```
+Discover valid values: `list(node.par.outputresolution.menuNames)`
+
+### 23. Large GLSL shaders break curl JSON escaping
+
+GLSL code full of single/double quotes, backslashes, and special chars will corrupt the JSON payload when sent via `curl -d`. **Write the shader to a temp file and load it in TD:**
+```python
+# Agent side: write shader to /tmp/shader.glsl via write_file
+# TD side (via td_exec):
+sd = root.create(textDAT, 'shader_code')
+with open('/tmp/shader.glsl', 'r') as f:
+    sd.text = f.read()
+```
+This avoids all escaping issues. The TD Python environment has full filesystem access.
+
+### 24. TD crashes lose everything — the WebServer DAT must be re-deployed
+
+If TD crashes (common with heavy GLSL or rapid-fire API calls), all nodes including the WebServer DAT are lost. On relaunch, port 9981 is dead. Recovery:
+1. Detect: `curl` returns exit code 7 (connection refused) or `lsof -i :9981` shows nothing
+2. Check: `pgrep TouchDesigner` to confirm TD is running
+3. Re-deploy: user must paste `exec(open('...custom_api_handler.py').read())` into Textport again
+4. Verify: poll port 9981 until API responds
+
+The `td_exec()` helper should handle this gracefully:
+```python
+def td_exec(script):
+    escaped = json.dumps({"script": script})
+    cmd = f"curl -s --max-time 15 -X POST -H 'Content-Type: application/json' -d {shlex.quote(escaped)} 'http://127.0.0.1:9981/api/td/server/exec'"
+    r = terminal(cmd, timeout=20)
+    if r.get('exit_code') == 7:
+        return {'error': 'TD not responding — WebServer DAT may need re-deploy'}
+    try:
+        return json.loads(r['output'])
+    except:
+        return {'error': 'Bad response', 'raw': r['output'][:200]}
+```