Concepts

Agnify processes video through a configurable graph of inference and data operations. This page covers the core model, the available node types, and how to work with them.

Execution, Pipeline, Template

Execution

An execution is the primary object. It binds a pipeline topology to a source (a video file, multiple files or a connected camera) and tracks the run lifecycle: draft, running, completed, or failed.

Two modes:

• Bounded: runs to completion over a finite source (uploaded video, camera playback date range). Produces MP4, detailed replay in Rerun format, and results.json on completion.
• Unbounded: runs continuously over a live source (camera, open-ended file stream). Emits events and a live preview frame; stops when you stop it.

A new execution starts as a draft. You configure the pipeline on the draft before running. Once started, the pipeline config is frozen.

Pipeline

A pipeline is a DAG of nodes stored on an execution. Nodes pass data through named topics: a detection node writes to masks, a tracker reads masks and writes tracks, and so on. The topology is a snapshot on the execution, not a reference to a shared template.

Template

A template is a saved pipeline topology with default parameter values. Instantiating a template generates fresh node IDs and patches the source type to match what you chose (uploaded video or camera). The analysis layer (detection, counting, VLM) is source-agnostic across template instances.

Built-in templates cover common cases. You can save any execution as a template from UI by clicking “Save as template” or by using the CLI:

agnify templates save <execution_id> <name>

Nodes

The canvas palette lists available node types. Each node has a typed parameter schema; the config panel on the right generates the form automatically when you select a node.

Sources

A pipeline starts with one source. Every source is followed by a FrameExtractor, which controls the sampling rate (interval_ms), crop region, and an optional change-detection gate that skips frames where the scene has not changed significantly. Enabling change detection reduces inference calls on footage with long static stretches.

Node	Use
VideoSource	Uploaded video file
CameraSource	Camera footage or S3 prefix, with an optional date range

Detection

Detection nodes consume frames and emit bounding boxes or masks to the masks topic.

Node	Use
DFineInference	Fast bounding-box detection. Supports COCO classes and fine-tuned models.
SAM3Inference	Open-vocabulary segmentation driven by a text prompt (e.g. “red bottle”). Slower than D-FINE; use when you need pixel masks rather than boxes.
PoseEstimation	COCO-17 skeleton keypoints. Requires DFineInference upstream.

Analysis

Node	Use
Tracker	Multi-object tracking across frames. Emits finished tracks with full history. Pairs with AttributeExtractor. Cannot be combined with ObjectCounter on the same masks topic.
ObjectCounter	Counts objects crossing a line or entering a polygon. Cannot be combined with Tracker on the same masks topic.
AttributeExtractor	Runs a VLM call per finished track to extract structured attributes (color, type, plate number, etc.). Fires when the tracker prunes a track, so results are per object and arrive after the track ends.
ModelInference	Free-form VLM prompt on every frame. Use when you want per-frame description rather than per-object attributes. Can return structured output instead of a sentence — switch it on, list the fields you want (true/false, a number, one of a fixed set), and every frame answers with exactly those, validated, ready for a chart or a Custom Python rule. Can also carry reference images — a few labelled example images attached to every call, so the prompt can ask “does this match the good crust reference?” instead of describing the target in words.
RunSynthesis	Reads everything the other nodes reported across the whole video and answers once, at the end — a summary, or the same structured fields, for the run as a whole. See Run Synthesis below.
CustomPython	Runs your Python per frame to aggregate the analysis topics over time into custom events, metrics, and alerts — the stateful logic (trigger on change, threshold over a window) the fixed nodes can’t express. See Custom Python below.
Redactor	Blurs or fills masked regions before they reach downstream nodes. Incompatible with the FrameExtractor change-detection option.

Outputs

Node	Use
VideoOutput	Annotated MP4 with detection and segmentation overlays
RerunOutput	Rerun recording (.rrd) for frame-by-frame scrubbing in the Rerun viewer
MetricsChannel	Tags events with a channel name for cross-execution aggregation on the dashboard

Run Synthesis

Every other node answers frame by frame. A RunSynthesis node answers once per run: it collects what the upstream nodes reported — the per-frame descriptions, the activity timeline, detection totals, counts, per-object attributes — and sends that as text to a language model at the end of the video.

Use it when the deliverable is one answer about the whole video rather than a stream of per-frame results: “summarise what the camera saw”, “did the machine malfunction, or did someone stop it for a normal reason?”, “which step did it get stuck on?”

Prompt — what you want answered. The run’s collected observations are appended below it automatically.

Structured output (optional) — the same switch and field editor as Frame Description: turn it on and list the fields the answer must carry (text, yes/no, a number, one of a fixed set) and the answer comes back as exactly those, validated. Leave it off for a free-form paragraph.

Where it shows up — on the run’s Result tab, above the rest of the results, with an expandable view of exactly what the model read. It’s also downloadable as JSON.

Two things to know: it only knows what the other nodes reported, so add the node that observes what your question needs; and it needs a video with an end, so it isn’t available on live camera streams.

Custom Python

A CustomPython node runs a small Python snippet once per frame over the upstream outputs and turns them into custom events, metric series, and Telegram alerts — adding stateful logic like “alert only when the count changes” or “fire if more than N for 30 s” that the fixed nodes don’t cover on their own. The code runs in a sandbox with no filesystem, network, or imports; it reaches the outside world only through the host functions below.

Inputs — pick a subset of masks / counter / tracks / attributes / results. Each frame the code gets wave: wave.detections (each with .label, .score, .bbox, .object_id), wave.count, wave.inference (the VLM text), wave.structured (the VLM’s fields when it returns structured output — wave.structured.get("no_sauce")), wave.attrs, and wave.finished (tracks that ended this frame).

Host functions (the only egress) — emit_event(name, fields), emit_metric(name, value), notify(text) (Telegram), get_state(key, default) / set_state(key, value) (persist across frames within a run), recent(seconds), count_by / group_by, now_ms, log.

Supported Python — def, comprehensions, f-strings, try/except. No import, class, with, match, async, or third-party libraries. The editor validates your code live.

Output — name the node’s output_topic (default custom); events record as <topic>.<name> and metrics as <topic>.metric. A Telegram node’s Notify on can forward this topic’s notify() messages, and metrics plot as time-series in the Rerun viewer.

Example — notify only when the object count changes (Inputs = counter):

prev = get_state("count", -1)
if wave.count != prev:
    notify(f"count changed: {prev} -> {wave.count}")
    set_state("count", wave.count)
emit_metric("count", wave.count)

Each frame is capped (≤ 20 events, ≤ 1 notify, 50 ms / 16 MiB); on error the frame is skipped and the run continues.

Creating a pipeline

From a template

1. Click New execution on the Executions page.
2. Select a template. The canvas opens with the pre-wired topology.
3. Set the source: attach a file to VideoSource, or configure CameraSource with a camera and date range.
4. Adjust parameters in the config panel as needed.
5. Click Run.

From scratch

1. Click New execution, then choose to start without a template.
2. Add nodes from the canvas palette.
3. Connect them by dragging edges.
4. Configure each node in the config panel.
5. Click Run.

Minimum viable pipeline: source, FrameExtractor, VideoOutput. Add detection and analysis nodes between FrameExtractor and the output.

Running and editing

Run: validates the topology, checks the credit balance, and starts the wave executor.

Stop: unbounded runs stop via the Stop button in the execution header. Bounded runs stop automatically when the source is exhausted.

Edit after run: the pipeline config freezes when a run starts. To change the topology, create a new draft and configure it before running. Starting from a template or copying an existing execution carries over the pipeline snapshot as a starting point.

Artifacts: after a bounded run finishes, the MP4, Rerun recording, and results.json appear on the execution detail page as download links.

Credits: each inference call deducts credits from the organization balance. A run will not start if the balance is at zero or below. Top up at any time from Settings.

Using the agent

The agent takes a text description of what you want to detect or analyze and builds a pipeline draft. It reads the video thumbnail, picks the appropriate nodes, sets defaults, and writes the DAG. Its edits appear on the canvas as it works.

To start a session: open the agent drawer (chat icon) on a draft execution that has a video attached, then describe your goal.

Example prompts:

• “Count people crossing the entrance line”
• “Extract the license plate from each tracked vehicle”
• “Detect dropped packages on the conveyor belt”

When the agent finishes, review the topology, adjust any parameters, and click Run yourself. The agent does not start runs.

If you edit a node directly while the agent is working, the agent reads your changes on its next pipeline fetch. Last write wins.

The agent is limited to bounded video-upload runs. Camera sources and streaming pipelines are not supported in v1.

CLI

The same operations are available from the CLI (agnify login first):

# List available node types
agnify nodes list

# Get the full schema and hints for one node type
agnify nodes schema DFineInference

# List saved templates
agnify templates list

# Fetch, validate, or update a pipeline draft
agnify pipeline get
agnify pipeline validate '<json>'
agnify pipeline set '<json>'

# Get the first frame of an uploaded file as a local PNG path
agnify file frame <file_id>

These are the same commands the in-browser agent uses internally.