Describing the task
You start by telling the agent in a chat what you need labeled. For example, “classify these support tickets by intent” or “draw bounding boxes around every cat in these images.” You don’t need to assemble or configure the pipeline by hand. From that description the agent assembles a pipeline: a sequence of nodes that takes your data in and produces labeled results. It decides which kinds of nodes the task needs, sets up the instructions and the labeling screen the experts will see, and sets the expert audience (languages, specializations, countries) and the pricing. When the data format, the languages your labelers should speak, or a quality threshold is ambiguous, it asks you rather than guessing. Ask the agent to change any part of the proposed pipeline that does not match your task.What a pipeline is made of
A pipeline is a connected set of nodes. Each node is one of a small, fixed set of kinds, and the agent picks the kind that fits each part of your task:- Generation node (reference). One labeler per item: annotating images, writing or rewriting text, rating or comparing outputs, or checking the output of a model. Expert labelers do this work by default, and it is the only node that can hand the same job to an LLM instead.
- Quorum node (reference). The same item is labeled independently by several experts (you choose how many), and you get back all of their answers, so you can see where they agree and where they disagree. It does not decide a winner for you. If you want to reduce the answers to a single value, a following Code node computes it.
- Code node (reference). A deterministic transformation: parsing, filtering, math, or calling an external service. Use this when the answer can be computed from the input without human judgment.
- Match node (reference). Routes items down different paths based on their content, so different items get handled differently.
How quality is controlled
Quality control is on by default on every labeling node. Which reviewer checks the labels, and for which items, is the node’s review path. On a Generation node, ask the agent to enable human review if needed. See Quality control.The self-check
Use Code Test for sample code inputs, Annotation Quality Assurance for sample annotations, and Synthetic Generation for LLM-generated samples. Preview the labeling interface and test a submission before deploying. Configuration errors and failed checks block deployment. Unrun test checks are warnings you can review and override. The built-in lints start automatically after edits and must finish before deployment: Quality Requirements and User Interface Types on labeling nodes, Dataset Items on Start, and Exam Golden Dataset on expert-labeling nodes with an enabled exam. See Self-check for test cases, lint findings, and the Issues window.Launching a run
Real labeling costs money, so the decision to spend is always yours. When the pipeline is built and free of errors, you launch the run. The agent does not start it for you. If you want to bound the cost, you can set a cap on how many items the run processes, so you label a subset rather than the whole dataset. A single run launches at most 10,000 items. For expert labeling, your charge is the per-task price you set plus the platform commission.Results
While a run is going, you can track its progress and look at individual results as they come in. When it finishes, you export the labeled dataset.One pipeline, one run
Deploying locks the pipeline: you cannot edit it, and you cannot run it a second time. The one exception is a failed deploy, which frees the same pipeline to fix and deploy again. To label another batch, clone the pipeline, re-point the copy, and deploy again.
Related
- Get your first labeled dataset — build and run a pipeline end to end.
- Fields and data — the data model a pipeline reads and writes.
- Quality control — how a node’s labels are reviewed.
- Pipeline nodes — what each kind of node does.