hermes-agent/optional-skills/creative/comfyui/SKILL.md

name	description	version	requires	author	license	platforms	prerequisites	setup	metadata
comfyui	Generate images, video, and audio with ComfyUI — install, launch, manage nodes/models, run workflows with parameter injection. Uses the official comfy-cli for lifecycle and direct REST API for execution.	4.0.0	ComfyUI (local or Comfy Cloud); comfy-cli (pip install comfy-cli)	kshitijk4poor alt-glitch	MIT	macos linux windows	commands python3	help pip install comfy-cli && comfy install. Cloud: get API key at platform.comfy.org	hermes tags related_skills category comfyui image-generation stable-diffusion flux creative generative-ai video-generation stable-diffusion-image-generation image_gen creative

ComfyUI

Generate images, video, and audio through ComfyUI using the official comfy-cli for setup/management and direct REST API calls for workflow execution.

Reference files in this skill:

• references/official-cli.md — comfy-cli command reference (install, launch, nodes, models)
• references/rest-api.md — ComfyUI REST API endpoints (local + cloud)
• references/workflow-format.md — workflow JSON format, common node types, parameter mapping

Scripts in this skill:

• scripts/comfyui_setup.sh — full setup automation (install + launch + verify)
• scripts/extract_schema.py — reads workflow JSON, outputs which parameters are controllable
• scripts/run_workflow.py — injects user args, submits workflow, monitors progress, downloads outputs
• scripts/check_deps.py — checks if required custom nodes and models are installed

When to Use

• User asks to generate images with Stable Diffusion, SDXL, Flux, or other diffusion models
• User wants to run a specific ComfyUI workflow
• User wants to chain generative steps (txt2img → upscale → face restore)
• User needs ControlNet, inpainting, img2img, or other advanced pipelines
• User asks to manage ComfyUI queue, check models, or install custom nodes
• User wants video/audio generation via AnimateDiff, Hunyuan, AudioCraft, etc.

Architecture: Two Layers

┌─────────────────────────────────────────────────────┐
│ Layer 1: comfy-cli (official)                       │
│   Setup, lifecycle, nodes, models                   │
│   comfy install / launch / stop / node / model      │
└─────────────────────────┬───────────────────────────┘
                          │
┌─────────────────────────▼───────────────────────────┐
│ Layer 2: REST API + skill scripts                   │
│   Workflow execution, param injection, monitoring   │
│   POST /api/prompt, GET /api/view, WebSocket        │
│   scripts/run_workflow.py, extract_schema.py        │
└─────────────────────────────────────────────────────┘

Why two layers? The official CLI handles installation and server management excellently but has minimal workflow execution support (just raw file submission, no param injection, no structured output). The REST API fills that gap — the scripts in this skill handle the param injection, execution monitoring, and output download that the CLI doesn't do.

Quick Start

Detect Environment

# What's available?
command -v comfy >/dev/null 2>&1 && echo "comfy-cli: installed"
curl -s http://127.0.0.1:8188/system_stats 2>/dev/null && echo "server: running"

If nothing is installed, go to Setup & Onboarding below. If the server is already running, skip to Core Workflow.

Core Workflow

Step 1: Get a Workflow

Users provide workflow JSON files. These come from:

• ComfyUI web editor → "Save (API Format)" button
• Community downloads (civitai, Reddit, Discord)
• The scripts/ directory of this skill (example workflows)

The workflow must be in API format (node IDs as keys with class_type). If user has editor format (has nodes[] and links[] at top level), they need to re-export using "Save (API Format)" in the ComfyUI web editor.

Step 2: Understand What's Controllable

python3 scripts/extract_schema.py workflow_api.json

Output (JSON):

{
  "parameters": {
    "prompt": {"node_id": "6", "field": "text", "type": "string", "value": "a cat"},
    "negative_prompt": {"node_id": "7", "field": "text", "type": "string", "value": "bad quality"},
    "seed": {"node_id": "3", "field": "seed", "type": "int", "value": 42},
    "steps": {"node_id": "3", "field": "steps", "type": "int", "value": 20},
    "width": {"node_id": "5", "field": "width", "type": "int", "value": 512},
    "height": {"node_id": "5", "field": "height", "type": "int", "value": 512}
  }
}

Step 3: Run with Parameters

Local:

python3 scripts/run_workflow.py \
  --workflow workflow_api.json \
  --args '{"prompt": "a beautiful sunset over mountains", "seed": 123, "steps": 30}' \
  --output-dir ./outputs

Cloud:

python3 scripts/run_workflow.py \
  --workflow workflow_api.json \
  --args '{"prompt": "a beautiful sunset", "seed": 123}' \
  --host https://cloud.comfy.org \
  --api-key "$COMFY_CLOUD_API_KEY" \
  --output-dir ./outputs

Step 4: Present Results

The script outputs JSON with file paths:

{
  "status": "success",
  "outputs": [
    {"file": "./outputs/ComfyUI_00001_.png", "node_id": "9", "type": "image"}
  ]
}

Show images to the user via vision_analyze or return the file path directly.

Decision Tree

User says	Tool	Command
"install ComfyUI"	comfy-cli	comfy install
"start ComfyUI"	comfy-cli	comfy launch --background
"stop ComfyUI"	comfy-cli	comfy stop
"install X node"	comfy-cli	comfy node install <name>
"download X model"	comfy-cli	comfy model download --url <url>
"list installed models"	comfy-cli	comfy model list
"list installed nodes"	comfy-cli	comfy node show installed
"generate an image"	script	run_workflow.py --args '{"prompt": "..."}'
"use this image" (img2img)	REST	upload image, then run_workflow.py
"what can I change in this workflow?"	script	extract_schema.py workflow.json
"check if workflow deps are met"	script	check_deps.py workflow.json
"what's in the queue?"	REST	curl http://HOST:8188/queue
"cancel that"	REST	curl -X POST http://HOST:8188/interrupt
"free GPU memory"	REST	curl -X POST http://HOST:8188/free

Setup & Onboarding

When a user asks to set up ComfyUI, walk them through the path that fits their situation. Ask what hardware they have and whether they want local or cloud.

Official docs: https://docs.comfy.org/installation CLI docs: https://docs.comfy.org/comfy-cli/getting-started Cloud docs: https://docs.comfy.org/get_started/cloud

Choosing an Installation Path

Situation	Recommended Path
No GPU / just want to try it	Comfy Cloud (zero setup)
Windows + NVIDIA GPU + non-technical	ComfyUI Desktop (one-click installer)
Windows + NVIDIA GPU + technical	Portable build or comfy-cli
Linux + any GPU	comfy-cli (easiest) or manual install
macOS + Apple Silicon	ComfyUI Desktop (macOS) or comfy-cli
Headless / server / CI	comfy-cli

---

Path A: Comfy Cloud (No Local Install)

For users without a capable GPU or who want zero setup. Powered by RTX 6000 Pro GPUs, all models pre-installed.

Docs: https://docs.comfy.org/get_started/cloud

1. Go to https://comfy.org/cloud and sign up
2. Get an API key at https://platform.comfy.org/login
   • Click + New in API Keys section → Generate
   • Save immediately (only visible once)
3. Set the key:

   export COMFY_CLOUD_API_KEY="comfyui-xxxxxxxxxxxx"
   

4. Run workflows via the script or web UI:

   python3 scripts/run_workflow.py \
     --workflow workflow_api.json \
     --args '{"prompt": "a cat"}' \
     --host https://cloud.comfy.org \
     --api-key "$COMFY_CLOUD_API_KEY" \
     --output-dir ./outputs
   

Pricing: https://www.comfy.org/cloud/pricing Subscription required. Concurrent limits: Free/Standard: 1 job, Creator: 3, Pro: 5.

---

Path B: ComfyUI Desktop (Windows/macOS)

One-click installer for non-technical users. Currently Beta.

Docs: https://docs.comfy.org/installation/desktop

• Windows (NVIDIA): https://download.comfy.org/windows/nsis/x64
• macOS (Apple Silicon): Available from https://comfy.org (download page)

Steps:

1. Download and run installer
2. Select GPU type (NVIDIA recommended, or CPU mode)
3. Choose install location (SSD recommended, ~15GB needed)
4. Optionally migrate from existing ComfyUI Portable install
5. Desktop launches automatically — web UI opens in browser

Desktop manages its own Python environment. For CLI access to the bundled env:

cd <install_dir>/ComfyUI
.venv/Scripts/activate   # Windows
# or use the built-in terminal in the Desktop UI

Limitations: Desktop uses stable releases (may lag behind latest). Linux not supported for Desktop — use comfy-cli or manual install.

---

Path C: ComfyUI Portable (Windows Only)

Standalone package with embedded Python. Extract and run. No install.

Docs: https://docs.comfy.org/installation/comfyui_portable_windows

1. Download from https://github.com/comfyanonymous/ComfyUI/releases
   • Standard: Python 3.13 + CUDA 13.0 (modern NVIDIA GPUs)
   • Alt: PyTorch CUDA 12.6 + Python 3.12 (NVIDIA 10 series and older)
   • AMD (experimental)
2. Extract with 7-Zip
3. Run run_nvidia_gpu.bat (or run_cpu.bat)
4. Wait for "To see the GUI go to: http://127.0.0.1:8188"

Update: run update/update_comfyui.bat (latest commit) or update/update_comfyui_stable.bat (latest stable release).

---

Path D: comfy-cli (All Platforms — Recommended for Agents)

The official CLI is the best path for headless/automated setups.

Docs: https://docs.comfy.org/comfy-cli/getting-started Repo: https://github.com/Comfy-Org/comfy-cli

Prerequisites

• Python 3.10+ (3.13 recommended)
• pip (or conda/uv)
• GPU drivers installed (CUDA for NVIDIA, ROCm for AMD)

Install comfy-cli

pip install comfy-cli
# or
uvx --from comfy-cli comfy --help

Disable analytics (avoids interactive prompt):

comfy --skip-prompt tracking disable

Install ComfyUI

# Interactive (prompts for GPU type)
comfy install

# Non-interactive variants:
comfy --skip-prompt install --nvidia              # NVIDIA (CUDA)
comfy --skip-prompt install --amd                 # AMD (ROCm, Linux)
comfy --skip-prompt install --m-series            # Apple Silicon (MPS)
comfy --skip-prompt install --cpu                 # CPU only (slow)

# With faster dependency resolution:
comfy --skip-prompt install --nvidia --fast-deps

Default location: ~/comfy/ComfyUI (Linux), ~/Documents/comfy/ComfyUI (macOS/Win). Override with: comfy --workspace /custom/path install

Launch Server

comfy launch --background              # background daemon on :8188
comfy launch                           # foreground (see logs)
comfy launch -- --listen 0.0.0.0       # accessible on LAN
comfy launch -- --port 8190            # custom port
comfy launch -- --lowvram              # low VRAM mode (6GB cards)

Verify server is running:

curl -s http://127.0.0.1:8188/system_stats | python3 -m json.tool

Stop background server:

comfy stop

---

Path E: Manual Install (Advanced / All Hardware)

For full control or unsupported hardware (Ascend NPU, Cambricon MLU, Intel Arc).

Docs: https://docs.comfy.org/installation/manual_install GitHub: https://github.com/comfyanonymous/ComfyUI

# 1. Create environment
conda create -n comfyenv python=3.13
conda activate comfyenv

# 2. Clone
git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI

# 3. Install PyTorch (pick your hardware)
# NVIDIA:
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu130
# AMD (ROCm 6.4):
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.4
# Apple Silicon:
pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu
# Intel Arc:
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/xpu
# CPU only:
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu

# 4. Install ComfyUI deps
pip install -r requirements.txt

# 5. Run
python main.py
# With options: python main.py --listen 0.0.0.0 --port 8188

---

Post-Install: Download Models

ComfyUI needs at least one checkpoint model to generate images.

Using comfy-cli:

# SDXL (general purpose, ~6.5GB)
comfy model download \
  --url "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors" \
  --relative-path models/checkpoints

# SD 1.5 (lighter, ~4GB, good for low VRAM)
comfy model download \
  --url "https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.safetensors" \
  --relative-path models/checkpoints

# From CivitAI (may need API token):
comfy model download \
  --url "https://civitai.com/api/download/models/128713" \
  --relative-path models/checkpoints \
  --set-civitai-api-token "YOUR_TOKEN"

# LoRA adapters:
comfy model download --url "<URL>" --relative-path models/loras

Manual download: Place .safetensors / .ckpt files directly into the ComfyUI/models/checkpoints/ directory (or loras/, vae/, etc.).

List installed models:

comfy model list

---

Post-Install: Install Custom Nodes

Custom nodes extend ComfyUI's capabilities (upscaling, video, ControlNet, etc.).

comfy node install comfyui-impact-pack           # popular utility pack
comfy node install comfyui-animatediff-evolved    # video generation
comfy node install comfyui-controlnet-aux         # ControlNet preprocessors
comfy node install comfyui-essentials             # common helpers
comfy node update all                            # update all nodes

Check what's installed:

comfy node show installed

Install deps for a specific workflow:

comfy node install-deps --workflow=workflow_api.json

---

Post-Install: Verify Setup

# Check server is responsive
curl -s http://127.0.0.1:8188/system_stats | python3 -m json.tool

# Check a workflow's dependencies
python3 scripts/check_deps.py workflow_api.json --host 127.0.0.1 --port 8188

# Test a generation
python3 scripts/run_workflow.py \
  --workflow workflow_api.json \
  --args '{"prompt": "test image, high quality"}' \
  --output-dir ./test-outputs

Image Upload (img2img / Inpainting)

Upload files directly via REST:

# Upload input image
curl -X POST "http://127.0.0.1:8188/upload/image" \
  -F "image=@photo.png" -F "type=input" -F "overwrite=true"
# Returns: {"name": "photo.png", "subfolder": "", "type": "input"}

# Upload mask for inpainting
curl -X POST "http://127.0.0.1:8188/upload/mask" \
  -F "image=@mask.png" -F "type=input" \
  -F 'original_ref={"filename":"photo.png","subfolder":"","type":"input"}'

Then reference the uploaded filename in workflow args:

python3 scripts/run_workflow.py --workflow inpaint.json \
  --args '{"image": "photo.png", "mask": "mask.png", "prompt": "fill with flowers"}'

Cloud Execution

Base URL: https://cloud.comfy.org Auth: X-API-Key header

# Submit workflow
python3 scripts/run_workflow.py \
  --workflow workflow_api.json \
  --args '{"prompt": "cyberpunk city"}' \
  --host https://cloud.comfy.org \
  --api-key "$COMFY_CLOUD_API_KEY" \
  --output-dir ./outputs \
  --timeout 300

# Upload image for cloud workflows
curl -X POST "https://cloud.comfy.org/api/upload/image" \
  -H "X-API-Key: $COMFY_CLOUD_API_KEY" \
  -F "image=@input.png" -F "type=input" -F "overwrite=true"

Concurrent job limits:

Tier	Concurrent Jobs
Free/Standard	1
Creator	3
Pro	5

Extra submissions queue automatically.

Queue & System Management

# Check queue
curl -s http://127.0.0.1:8188/queue | python3 -m json.tool

# Clear pending queue
curl -X POST http://127.0.0.1:8188/queue -d '{"clear": true}'

# Cancel running job
curl -X POST http://127.0.0.1:8188/interrupt

# Free GPU memory (unload all models)
curl -X POST http://127.0.0.1:8188/free -H "Content-Type: application/json" \
  -d '{"unload_models": true, "free_memory": true}'

# System stats (VRAM, RAM, GPU info)
curl -s http://127.0.0.1:8188/system_stats | python3 -m json.tool

Pitfalls

1. API format required — comfy run and the scripts only accept API-format workflow JSON. If the user has editor format (from "Save" not "Save (API Format)"), they need to re-export. Check: API format has class_type in each node object, editor format has top-level nodes and links arrays.

2. Server must be running — All execution requires a live server. comfy launch --background starts one. Check with curl http://127.0.0.1:8188/system_stats.

3. Model names are exact — Case-sensitive, includes file extension. Use comfy model list to discover what's installed.

4. Missing custom nodes — "class_type not found" means a required node isn't installed. Run check_deps.py to find what's missing, then comfy node install <name>.

5. Working directory — comfy-cli auto-detects the ComfyUI workspace. If commands fail with "no workspace found", use comfy --workspace /path/to/ComfyUI <command> or comfy set-default /path/to/ComfyUI.

6. Cloud vs local output download — Cloud /api/view returns a 302 redirect to a signed URL. Always follow redirects (curl -L). The run_workflow.py script handles this automatically.

7. Timeout for video/audio — Long generations (video, high step counts) can take minutes. Pass --timeout 600 to run_workflow.py. Default is 120 seconds.

8. tracking prompt — First run of comfy may prompt for analytics tracking consent. Use comfy --skip-prompt tracking disable to skip it non-interactively.

9. comfy-cli invocation via uvx — If comfy-cli is not installed globally, invoke with uvx --from comfy-cli comfy <command>. All examples in this skill use bare comfy but prepend uvx --from comfy-cli if needed.

Verification Checklist

• comfy available on PATH (or uvx --from comfy-cli comfy --help works)
• curl http://127.0.0.1:8188/system_stats returns JSON
• comfy model list shows at least one checkpoint
• Workflow JSON is in API format (has class_type keys)
• check_deps.py reports no missing nodes/models
• Test run completes and outputs are saved