Prepare training data from labels

This guide converts the result of one Data Labeling pipeline run into the chat JSONL that Training reads. The worked mapping is for a classification pipeline with one text input named ticket_text and one string output named intent. The conversion creates three files:

training.jsonl contains the rows you upload for Training.
evaluation-inputs.jsonl contains prompts to send to the trained model later.
evaluation-gold.jsonl contains the expected labels and stays local.

Upload only training.jsonl. Uploading either evaluation file, or copying its labels into the training rows, leaks the answers you intend to use for final evaluation.

Before you start

You need:

A completed pipeline run whose End node returns the label you want the model to learn.
The run’s JSON export. Follow Export your labeled results and wait for the run to finish before starting the export.
Python 3.9 or later for the local conversion.
The exact input and output field names in the export. See Export format if you need to inspect the entry shape.

Use one run export at a time. The index is unique within a run, but it is not a global item identifier across several runs.

Map the export to messages

For this classification example, each completed export entry maps as follows:

Export value	Training or evaluation value	Purpose
`input.ticket_text`	A `user` message’s `content`	The text the model sees
`output.intent`	The final `assistant` message’s `content`	The answer the model learns
`index`	The evaluation join key	Matches a prediction to its held-out answer
`status`	A filter	Only `completed` entries are eligible
`metrics`	Not copied	Timing, attempts, and annotator identifiers are not model inputs

A completed entry can look like this:

{
  "index": 41,
  "status": "completed",
  "input": { "ticket_text": "I was charged twice for one subscription." },
  "output": { "intent": "complaint" }
}

It becomes this training row when it is assigned to the training partition:

{"messages":[{"role":"user","content":"I was charged twice for one subscription."},{"role":"assistant","content":"complaint"}]}

Do not convert failed, cancelled, paused, in_progress, or idle entries. They do not represent a completed input-to-label pair. The script below also stops if a completed entry lacks the selected string input or label, rather than silently creating a partial example.

A Quorum output can contain per-expert arrays. Point --label-field at the resolved scalar output your End node returns for training. Do not stringify an answer array or use annotatorIds as the label unless that is deliberately the task you are training.

Save the converter

Save the following as prepare_label_export.py:

import argparse
import hashlib
import json
from pathlib import Path


def write_jsonl(path, rows):
    path.write_text(
        "".join(json.dumps(row, ensure_ascii=False) + "\n" for row in rows),
        encoding="utf-8",
    )


parser = argparse.ArgumentParser()
parser.add_argument("export", type=Path)
parser.add_argument("--input-field", required=True)
parser.add_argument("--label-field", required=True)
parser.add_argument("--evaluation-count", type=int, default=2)
parser.add_argument("--output-dir", type=Path, default=Path("prepared"))
args = parser.parse_args()

rows = json.loads(args.export.read_text(encoding="utf-8"))
if not isinstance(rows, list):
    raise SystemExit("The export must be a JSON array.")
if args.evaluation_count < 1:
    raise SystemExit("--evaluation-count must be at least 1.")

eligible = []
seen_indexes = set()
skipped_statuses = 0

for row in rows:
    if not isinstance(row, dict):
        raise SystemExit("Every export entry must be a JSON object.")
    if row.get("status") != "completed":
        skipped_statuses += 1
        continue

    index = row.get("index")
    input_values = row.get("input")
    output_values = row.get("output")
    if not isinstance(index, int):
        raise SystemExit("Every completed entry must have an integer index.")
    if index in seen_indexes:
        raise SystemExit(f"Duplicate completed index: {index}")
    seen_indexes.add(index)

    if not isinstance(input_values, dict) or not isinstance(output_values, dict):
        raise SystemExit(f"Completed entry {index} has no input or output object.")
    prompt = input_values.get(args.input_field)
    label = output_values.get(args.label_field)
    if not isinstance(prompt, str) or not prompt.strip():
        raise SystemExit(
            f"Completed entry {index} has no non-empty string input field "
            f"{args.input_field!r}."
        )
    if not isinstance(label, str) or not label.strip():
        raise SystemExit(
            f"Completed entry {index} has no non-empty string label field "
            f"{args.label_field!r}."
        )
    eligible.append((index, prompt.strip(), label.strip()))

if args.evaluation_count >= len(eligible):
    raise SystemExit("The evaluation count must leave at least one training row.")

groups = {}
for record in eligible:
    groups.setdefault(record[1], []).append(record)

ranked_groups = sorted(
    groups.values(),
    key=lambda group: (
        hashlib.sha256(group[0][1].encode("utf-8")).digest(),
        min(record[0] for record in group),
    ),
)
evaluation = []
for group in ranked_groups:
    if len(evaluation) >= args.evaluation_count:
        break
    evaluation.extend(group)

evaluation_indexes = {record[0] for record in evaluation}
training = [record for record in eligible if record[0] not in evaluation_indexes]
if len(training) < 10:
    raise SystemExit(
        "This split leaves fewer than 10 rows to upload for Training. Add completed labels or "
        "reduce --evaluation-count."
    )

training.sort(key=lambda record: record[0])
evaluation.sort(key=lambda record: record[0])
args.output_dir.mkdir(parents=True, exist_ok=True)

write_jsonl(
    args.output_dir / "training.jsonl",
    (
        {
            "messages": [
                {"role": "user", "content": prompt},
                {"role": "assistant", "content": label},
            ]
        }
        for _, prompt, label in training
    ),
)
write_jsonl(
    args.output_dir / "evaluation-inputs.jsonl",
    (
        {
            "index": index,
            "messages": [{"role": "user", "content": prompt}],
        }
        for index, prompt, _ in evaluation
    ),
)
write_jsonl(
    args.output_dir / "evaluation-gold.jsonl",
    (
        {"index": index, "expected": label}
        for index, _, label in evaluation
    ),
)

print(f"Eligible completed rows: {len(eligible)}")
print(f"Rows to upload for Training: {len(training)}")
print(f"Evaluation rows: {len(evaluation)}")
print(f"Skipped non-completed rows: {skipped_statuses}")

The split is reproducible. It ranks prompts by a SHA-256 digest and selects whole prompt groups for evaluation until it reaches the requested count. Exact duplicate prompts therefore cannot land in both partitions. Because duplicate prompts stay together, the actual evaluation count can be greater than --evaluation-count. The script requires at least 10 rows in training.jsonl after the local evaluation split. When you create a Training from the new dataset, the default 80/10/10 split assigns those 10 rows as 8 train, 1 validation, and 1 test. The 8 train rows fill the default batch size. Custom splits or larger batch sizes can require more rows.

Run the conversion

Run the converter with the field names from your export:

python3 prepare_label_export.py labeled-results.json \
  --input-field ticket_text \
  --label-field intent \
  --evaluation-count 2 \
  --output-dir prepared

A run with 14 eligible entries, unique prompts, and 2 unfinished entries prints:

Eligible completed rows: 14
Rows to upload for Training: 12
Evaluation rows: 2
Skipped non-completed rows: 2

Review the counts. An error naming a completed entry usually means a field name is wrong or the selected output is not a scalar string.

Inspect the separation

A line in evaluation-inputs.jsonl contains the prompt but not its answer:

{"index":41,"messages":[{"role":"user","content":"I was charged twice for one subscription."}]}

The matching line in evaluation-gold.jsonl keeps the answer separate:

{"index":41,"expected":"complaint"}

Keep evaluation-gold.jsonl away from the inference request. The index is only a local join key for scoring predictions later.

Upload the training partition

For this step, install the CLI, sign in, and select a project where your credentials can create and upload datasets. For browser login, select a team with Full access using the team switcher. For an API key, check CLI permissions. Create a dataset in the project selected by the CLI, upload only the training file, and copy the dataset ID printed by dataset list:

toloka dataset create support-ticket-training
toloka dataset upload support-ticket-training ./prepared/training.jsonl
toloka dataset list

Dataset uploads append rows. Do not repeat the upload to the same dataset unless you intend to add a second copy of every training example. The converter produces a messages array with one user turn and a final assistant label. For other conversation formats, see Training data requirements. A cost estimate validates the selected rows before a Training Run can start.

Keep the two evaluation boundaries separate

The local evaluation set stays separate from the dataset slice used by Training:

Keep evaluation-inputs.jsonl and evaluation-gold.jsonl out of Training.
Create the Training with --dataset pointing to the dataset containing only training.jsonl. The command creates an 80% train, 10% validation, 10% test split for this new dataset and freezes a slice.
The platform learns from the train part and reports validation metrics on the validation part. The test part stays out of learning.
Keep the local evaluation set untouched until you have actual predictions from the downloaded Model Version.

The same chat rows can be used for fine-tuning or gisting. Gisting additionally requires a prompt text file. The complete label-to-training-to-evaluation path is in Label, train, and evaluate.

Adapt the mapping

For a pipeline with several inputs, construct one user message that names each field consistently. For a generative output, put the exact target text in the final assistant message and choose an evaluation metric suited to that output. Do not copy export metadata into the conversation unless it is information the model should receive at inference time. If separate source items can contain semantically equivalent text with different wording, deduplicate or group them using a stable source identifier before conversion. The example only guarantees that identical prompt strings stay in one partition.

Start

Data Labeling

Train

Workflows

Admin

Prepare training data from labels

Before you start

Map the export to messages

Save the converter

Run the conversion

Inspect the separation

Upload the training partition

Keep the two evaluation boundaries separate

Adapt the mapping

​Before you start

​Map the export to messages

​Save the converter

​Run the conversion

​Inspect the separation

​Upload the training partition

​Keep the two evaluation boundaries separate

​Adapt the mapping

​Related

Before you start

Map the export to messages

Save the converter

Run the conversion

Inspect the separation

Upload the training partition

Keep the two evaluation boundaries separate

Adapt the mapping

Related