A testing framework for ComfyUI workflows. Provides custom nodes for defining tests and assertions, plus a CLI tool for running tests in CI/CD environments.
- Test Definition Nodes: Define test metadata, GPU requirements, and timeouts
- Assertion Nodes: Validate values, images, tensors, and execution state
- CLI Tool (
comfyci): Run test workflows against a ComfyUI instance - Perceptual Hashing: Compare images using difference hashing (dHash)
- CI/CD Ready: Designed for automated testing pipelines
Clone this repository into your ComfyUI custom nodes directory:
cd ComfyUI/custom_nodes
git clone https://github.com/Comfy-Org/ComfyUI-test-framework.gitRestart ComfyUI to load the test nodes.
The CLI tool uses uv for dependency management:
cd ComfyUI-test-framework/cli
# Run directly with uv
./comfyci.py --help
# Or install as a package
uv pip install -e .
comfyci --help| Node | Description |
|---|---|
| Test Definition | Define test metadata: name, description, GPU requirements, extra timeout |
| Node | Description |
|---|---|
| Assert Executed | Pass-through that verifies the upstream nodes executed (not cached) |
| Assert Equal | Check if two values are deeply equal |
| Assert Not Equal | Check if two values differ |
| Assert Image Match | Compare image against a perceptual hash with configurable threshold |
| Assert Contains Color | Verify image contains pixels of a specific color |
| Assert Tensor Shape | Validate tensor dimensions (batch, height, width, channels) |
| Assert Mask Coverage | Check that mask coverage (% of non-zero pixels) is within an expected range |
| Assert Mask Binary | Check that all mask values are exactly 0 or 1 (hard mask) |
| Assert Mask Fuzzy | Check that a soft-edge mask has a controlled amount of grey boundary pixels |
| Assert String Contains | Check that a string contains an expected substring |
| Assert String Not Contains | Check that a string does NOT contain a forbidden substring |
| Assert String Length | Check that a string's length is within min/max bounds |
| Assert String Match | Check that a string matches a pattern (exact or regex) |
| Assert In Range | Check if all tensor values are within min/max bounds |
| Node | Description |
|---|---|
| Test Image Generator | Generate test images: solid black, white, noise, face, synthetic benchmark pattern, or video benchmark pattern |
| Test Mask Generator | Generate test masks: solid, circle, gradient, checkerboard, noise, or half masks |
- Add a Test Definition node to your workflow
- Set the test name, description, and whether it requires GPU
- Connect your workflow outputs to assertion nodes
- Save the workflow as JSON
Example workflow structure:
[Load Image] → [Some Processing] → [Assert Image Match]
↘ [Assert Tensor Shape]
[Test Definition]
# Run all tests in a directory
comfyci ./tests/**/*.json --server localhost:8188
# Skip GPU-required tests
comfyci ./tests/**/*.json --cpu
# Verbose output
comfyci ./tests/**/*.json -v
# Stop on first failure
comfyci ./tests/**/*.json -x
# Run against Comfy Cloud
comfyci ./tests/**/*.json --cloud --op-entry Employee/TestCIThe Assert Image Match node uses perceptual hashing to compare images:
- Add the node without a hash - it will calculate and display the hash
- Click "Accept Hash" to copy the calculated hash into the input field
- Adjust the
deltathreshold (0.0 = exact match, 1.0 = any image) - Re-run to validate future executions against this baseline
Usage: comfyci [OPTIONS] PATTERNS...
Run ComfyUI test workflows.
PATTERNS: Glob patterns for test workflow JSON files (e.g., ./tests/**/*.json)
Options:
--server TEXT ComfyUI server address (host:port) [default: localhost:8188]
--ssl Use HTTPS/WSS for server connection
--cloud Use Comfy Cloud API (e.g., /jobs instead of /history)
--cpu Skip tests that require GPU
-v, --verbose Enable verbose output
-x, --failfast Stop on first test failure
--timeout INTEGER Override default timeout in seconds (default: 120 + test extraTime)
--no-color Disable colored output
--op-exe TEXT 1Password CLI executable (op or op.exe) [default: op.exe]
--op-entry TEXT 1Password entry path (e.g., Employee/TestCI). Fetches
email, password, FIREBASE_API_KEY.
--help Show this message and exit.
0: All tests passed1: One or more tests failed130: Interrupted by user (Ctrl+C)
MIT License - see LICENSE for details.