Skip to main content
The Quorum node has the same item labeled independently by one or more expert labelers and returns the full set of their labels, one label per expert. It computes no consensus: the labels come back as a raw array, and reducing them to one value, such as a majority vote, is the job of a following Code node that reads the array. Every setting described below can be set by asking the agent or edited by hand on the node’s cards until deploy. The one exception is the onboarding on/off switch, which only you control (see Onboarding). The other kinds of nodes a pipeline can contain are listed in Pipeline nodes.

​
Number of experts

Overlap is the number of experts who independently label each item to start with. Set a whole number from 1 to 100. The same starting count applies to every item and does not react to expert disagreement. Under LLM review, rejection can bring in additional experts beyond it (see When review rejects every answer). No-review mode requires at least 3 (see Quality control). A single run feeds the node at most 10,000 items — a platform-wide per-run cap (Label a large dataset in batches covers larger workloads). The overlap sets no cap of its own on the answers a run collects: each item starts at overlap answers, and escalation can add more. One expert means a single label per item, but the output shape does not change: outputs remain arrays, of one element or more if review rejections add answers. Verdicts stay per-expert arrays too, unlike the plain values the Generation node, the single-expert labeling node, returns. Both the expert-labeling charge and LLM review scale with this count: doubling the number of experts doubles both. A video field that a criterion reaches through field links, under LLM review, also bills a video-tool call per answer, so that scales with the overlap too.

​
Input and output fields

The node needs at least one field each expert reads (input) and at least one field each expert fills in (output). Input fields have no array shape because every expert reads the same item. Every output comes back as a per-expert array. See Fields and data for why. A repeating output is an array of arrays, with one inner array per expert. Each expert produces up to a declared maximum of 1 to 50 values. An optional output an expert leaves blank keeps its slot, filled with null. Leave the length of an output’s array open, because the number of answers varies from item to item. The deploy is blocked when an output’s type schema gives the array a minItems above 1 or a maxItems below the overlap (below twice the overlap under LLM review, see When review rejects every answer). It is also blocked when the array’s schema uses allOf, anyOf or oneOf: set length limits directly on the array and put other constraints in items, the schema of each answer. The array shape itself adds no cost. A repeating field checked by a quality criterion multiplies review calls (see Quality control).

​
Matching answers to experts

The node also returns a field named after the node with _annotator_ids at the end, for example rate_helpfulness_annotator_ids. It lists the ID of the expert who gave each answer. All output arrays of the node line up by position. The first ID belongs to the first answer in every answer field and every verdict field, the second ID to the second answer, and so on:
  • A rejected answer keeps its position. The verdict fields at the same position show what review decided.
  • One expert can hold more than one position in the same item. Do not assume that each position is a different person.
  • The field can be missing on an item. It is missing on items that finished before the field existed, and on canceled items that got no answers. A canceled item with some answers lists the experts of those answers.
  • The field is written to your dataset next to the answers. To use it inside the pipeline, add it as an input of the next Code node. To get it in the run export, add it to the outputs of the End node. Both cards add it as optional, because the node does not write it on every item.
For a canceled item, open Logs to inspect which submitted answers were accepted or rejected. Accepted answers are still billed, along with recorded review work and one-time per-expert charges. See What a canceled or failed item costs.

​
Instructions and the labeling interface

Experts see the written instructions, the labeling interface, and the display name set on the node’s Display name card. Instructions are free text and must not be empty. The display name is required and at most 64 characters. The labeling interface presents one expert’s view: each expert submits their own entry for every output field, with one value for a plain field, several for a repeating field, and none for an optional field they leave blank. These settings do not affect cost.

​
Quality criteria

The checks that define good work on this node. See the shared quality model for what a criterion is and the floor it clears to deploy. A criterion may target any output field, and a field may carry more than one criterion. Unlike a single-expert labeling node, not every output field needs one — but the node still needs at least one quality criterion to deploy, including under no review. Under no review the criteria are shown to experts as read-only guidance, produce no verdict fields, and add no cost. Under LLM review, each criterion adds review calls (see Quality control).

​
Quality control

How each expert’s work is checked: automated LLM review or no review (the card calls it No QA, Offboarding Only). This node has no human review. See the shared quality model for how review, verdicts, and the review metadata field work. When LLM review runs here, that review metadata field (_qa_metadata) is not guaranteed a value on every item. No review returns no verdicts and no review metadata field. Configure its offboarding rules before deployment. The _annotator_ids field stays available in both modes (see Matching answers to experts). No review requires all of:
  • at least one offboarding rule on the Offboarding card;
  • at least 3 experts per item;
  • a single-label output field — one where each expert picks exactly one label — for agreement to be measured on;
  • the per-expert task cap turned off.
LLM review is billed per call at the chosen model’s rate. The model is picked under Quality Assurance, or on this node by asking the agent (see the shared quality model). No review adds no review charge. In the cost estimate, the call count multiplies along these dimensions: The doubled review forecast covers additional answers after review rejection.

​
When review rejects every answer

Labeling starts at the configured overlap. If LLM review rejects every answer, the platform may collect another answer and review again, or the item may fail. Every collected answer stays in the output array, accepted or rejected. A completed item is not guaranteed to contain exactly overlap accepted answers, so downstream code must handle longer arrays and rejected entries. The forecast budgets twice the initial review-call count; it does not establish a runtime limit on the number of answers. Labeling starts at the configured overlap. If review rejects every answer, the platform may collect another answer and review again, or the item may fail. The forecast budgets twice the initial review-call count.

​
Offboarding rules

In No QA, Offboarding Only mode, add at least one offboarding rule on the Offboarding card. A Majority-vote agreement rule picks one single-label output field and offboards experts who persistently disagree with the majority on it. The field must be a top-level field where every expert picks one label. You can add multiple rules, but each field can appear in only one rule. This mode requires an overlap of at least 3 and the per-expert task cap off. Offboarding rules do not grade or reject individual answers, alter the returned answer arrays, or add a review charge.

​
Expert audience and pricing

Choose who labels the items and set the per-task price. Experts on this node are always ID-verified. A Quorum node always uses a labeling task. Data collection and the wider unverified pool are unavailable, while specializations, which require verified experts, are always selectable. You filter experts by languages, specializations, and countries. The audience card shows how many experts match the saved filters. When you ask the agent about the pool, it can count the same saved filters, so its number matches the card. The per-expert cap cannot be combined with no-review mode. Your expert-labeling charge is the per-task price you set plus the platform commission, billed for each expert per item. Expert Service Cost shows the combined charge, with the commission displayed as 33%. A tighter per-expert cap can increase the one-time onboarding and exam charges.

​
Onboarding

Experts complete this course before labeling items. The course is optional (on or off). Turning it on or off is yours alone, on the node’s Training card. The agent builds the course but never flips the switch. Deployment enforces the same per-block contract as the Generation node: quiz questions with named options and a marked answer, no empty blocks, and worked examples only on a built interface. See its onboarding contract. When on, the course must have:
  • 3 to 6 steps;
  • a quiz among them;
  • a stated role, goal, and success criteria for the expert.
A course can include practice exercises or worked examples, but cannot use both. Onboarding cannot be turned off when the node reuses the audience of a node that was deployed with onboarding (see Audience reuse). Onboarding is charged once per expert who joins the task. The cost breakdown shows Onboarding (one-time per expert), or Training (one-time per expert) when an exam is enabled. An enabled exam adds Exam (one-time per expert), charged when a passing expert starts work. The Exam card’s settings are described under Exam on the Generation node. See Pipeline pricing.

​
Audience reuse

Reusing the trained pool of experts from a node you deployed earlier lets returning experts skip onboarding. The reused node may be a past Quorum node or a past single-expert labeling node. The picker offers nodes from other pipelines of the same project. Ask the agent to select a deployed source node from any project your team owns. The two scopes are laid out under audience reuse on the Generation node. Only the labeling audience is reused because this node has no human-review stage to carry over. Experts who passed the earlier node’s onboarding receive tasks immediately. When the earlier node was deployed with onboarding, this node’s onboarding must stay on. A deployment with it off is rejected. Returning experts are not charged onboarding again, lowering the onboarding total.