Data labeling is the work of attaching the correct answer to raw examples (a box around a pedestrian, an intent on a support ticket, a speaker on an audio clip) so a model can learn from them or be tested against them. Whether your model works in production depends more on how consistent those labels are than on how many you have.
On this page
- What types of data labeling are there?
- How do you measure labeling quality?
- How should you handle ambiguous examples?
- What drives the cost of data labeling?
- When is a dataset ready to train on?
- What should you ask a data labeling vendor?
- When isn’t outsourcing labeling the right call?
- Frequently asked questions
Most labeling problems aren’t caused by careless annotators. They come from vague guidelines, unhandled edge cases and nobody checking agreement until the model underperforms. This guide covers what to measure, what drives cost, and how to tell when a dataset is ready to train on.
What types of data labeling are there?
The main families are image and video annotation (classification, bounding boxes, polygons, segmentation, keypoints, object tracking), text annotation (classification, entity recognition, intent, sentiment, relevance ranking), and audio annotation (transcription, speaker labels, sound events). LLM projects add a newer one: grading and ranking model outputs.
| Data type | Common tasks | Relative effort per item |
|---|---|---|
| Images | Classification, bounding boxes, polygons, segmentation, keypoints | Low (classification) to very high (pixel segmentation) |
| Video | Object tracking, action labels, event timestamps | High, because every frame or interval needs consistency |
| Text | Classification, named entities, intent, sentiment, relevance | Low to medium; higher when domain knowledge is needed |
| Audio | Transcription, speaker diarisation, sound events | Medium to high, roughly tied to audio length |
| LLM outputs | Pairwise preference, rubric grading, factuality checks | Medium to high; needs careful rubrics |
That last row matters more every year. If you’re building on large language models, your evaluation set is a labeled dataset too, and it deserves the same discipline.
How do you measure labeling quality?
Use three numbers: accuracy against a gold set that experts have labeled, agreement between annotators on the same items (Cohen’s or Fleiss’ kappa), and coverage of the edge cases your model will actually meet. Accuracy alone can look fine while two annotators quietly apply different rules.
- Gold-set accuracy. Seed a small, expert-labeled set into every batch and score annotators against it continuously, not once at onboarding.
- Inter-annotator agreement. Double-label a sample. Low agreement almost always points to an ambiguous guideline, not a bad annotator.
- Edge-case coverage. Track how many examples of each rare-but-important case you have. Models fail on the long tail.
- Class balance. Know your label distribution before training, so a 95% accuracy score isn’t just the majority class.
The target depends on the use. A content-tagging model can live with some noise. A model that flags safety or medical issues can’t, and should have expert review on every disputed item.
How should you handle ambiguous examples?
Give annotators a way to flag uncertainty instead of guessing, route flagged items to a reviewer, and turn every decision into a written rule with an example. After a few rounds the guideline stops being a one-page brief and becomes the most useful document in the project.
The pattern we follow on labeling work is simple. Annotators label a pilot batch, disagreements are reviewed together with the client’s domain expert, and the guideline is revised before the next batch. Skipping that loop is the most common reason a dataset has to be relabeled later.
What drives the cost of data labeling?
Four things: how long each item takes (a yes/no tag versus pixel-level segmentation), how much domain knowledge annotators need, how much review you require (single pass, double labeling or expert adjudication), and volume. Tooling and data security requirements add to it, especially if data can’t leave your environment.
We don’t quote per-item prices without seeing a sample, because the same task name (“bounding boxes”) can mean five seconds or two minutes per image depending on density and rules. The honest way to price it is a pilot: label a few hundred representative items, measure time per item and agreement, then estimate the full run from real numbers.
A few ways to keep cost down without hurting quality:
- Pre-label with an existing model and have humans correct, rather than label from scratch, once the model is decent.
- Spend review effort where disagreement is high, not uniformly.
- Label the examples the model is unsure about first (active learning) instead of labeling everything.
- Fix the guideline early. Relabeling is the most expensive line item there is.
When is a dataset ready to train on?
When the guideline has stopped changing, agreement on double-labeled samples is stable, the gold-set score meets your target, rare classes have enough examples to measure, and the data is versioned so you can trace a model back to the exact labels it used. If any of those is missing, you’re training on a moving target.
Readiness is also a data engineering question. You need repeatable ingestion, validation that catches broken files and impossible values, and dataset versioning. Our data engineering team builds those pipelines, and MLOps and AI infrastructure covers monitoring for drift once the model is live.
What should you ask a data labeling vendor?
Ask how they measure agreement and report it to you, who writes and maintains the guideline, how disputed items get resolved, where your data is stored and who can access it, and whether they’ll run a paid pilot before a volume commitment. Vague answers to the first question are the biggest warning sign.
- Do annotators work on your tool or theirs, and who owns the labeled output? (It should be you.)
- Are annotators employees or crowd workers? That changes training, confidentiality and consistency.
- How quickly can they add or remove people when volume changes?
- Can they handle your domain, or will your experts need to review everything?
When isn’t outsourcing labeling the right call?
When the labels need deep, scarce expertise, such as radiologists reading scans, or when data legally can’t leave your organisation. In those cases keep labeling in-house and outsource the tooling, pipelines and quality reporting around it instead.
Our data labeling services page describes how we run projects from Lalitpur, including the sample-batch pilot. If the labels feed an LLM product, our guide to building a GenAI MVP in 4–8 weeks shows where evaluation data fits in the schedule.
Frequently asked questions
How long does a labeling pilot take?
Usually one to two weeks for a few hundred to a thousand items, including at least one guideline revision. The pilot’s real output is a stable guideline and a measured time per item, not the labels themselves.
How do you handle sensitive data?
Agree it in writing before any data moves: where it’s stored, who can access it, whether it’s anonymised first, and how it’s deleted at the end. Where possible, annotators should work inside your environment so the data never leaves it.
