Arize-Annotation

# Arize Annotation Skill

> **`SPACE`** — All `--space` flags and the `ARIZE_SPACE` env var accept a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`.

This skill covers **annotation configs** (the label schema) and **annotation queues** (human review workflows), as well as programmatically annotating project spans via the Python SDK.

**Direction:** Human labeling in Arize attaches values defined by configs to **spans**, **dataset examples**, **experiment-related records**, and **queue items** in the product UI. This skill covers: `ax annotation-configs`, `ax annotation-queues`, and bulk span updates with `ArizeClient.spans.update_annotations`.

---

Prerequisites

Proceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront.

If an `ax` command fails, troubleshoot based on the error:

- `command not found` or version error → see references/ax-setup.md

- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys

- Space unknown → run `ax spaces list` to pick by name, or ask the user

- **Security:** Never read `.env` files or search the filesystem for credentials. Use `ax profiles` for Arize credentials and `ax ai-integrations` for LLM provider keys. If credentials are not available through these channels, ask the user.

---

Concepts

What is an Annotation Config?

An **annotation config** defines the schema for a single type of human feedback label. Before anyone can annotate a span, dataset record, experiment output, or queue item, a config must exist for that label in the space.

| Field | Description |

|-------|-------------|

| **Name** | Descriptive identifier (e.g. `Correctness`, `Helpfulness`). Must be unique within the space. |

| **Type** | `categorical` (pick from a list), `continuous` (numeric range), or `freeform` (free text). |

| **Values** | For categorical: array of `{"label": str, "score": number}` pairs. |

| **Min/Max Score** | For continuous: numeric bounds. |

| **Optimization Direction** | Whether higher scores are better (`maximize`) or worse (`minimize`). Used to render trends in the UI. |

Where labels get applied (surfaces)

| Surface | Typical path |

|---------|----------------|

| **Project spans** | Python SDK `spans.update_annotations` (below) and/or the Arize UI |

| **Dataset examples** | Arize UI (human labeling flows); configs must exist in the space |

| **Experiment outputs** | Often reviewed alongside datasets or traces in the UI — see arize-experiment, arize-dataset |

| **Annotation queue items** | `ax annotation-queues` CLI (below) and/or the Arize UI; configs must exist |

Always ensure the relevant **annotation config** exists in the space before expecting labels to persist.

---

Basic CRUD: Annotation Configs

List

ax annotation-configs list --space SPACE
ax annotation-configs list --space SPACE -o json
ax annotation-configs list --space SPACE --limit 20

Create — Categorical

Categorical configs present a fixed set of labels for reviewers to choose from.

ax annotation-configs create \
  --name "Correctness" \
  --space SPACE \
  --type categorical \
  --value correct \
  --value incorrect \
  --optimization-direction maximize

Common binary label pairs:

- `correct` / `incorrect`

- `helpful` / `unhelpful`

- `safe` / `unsafe`

- `relevant` / `irrelevant`

- `pass` / `fail`

Create — Continuous

Continuous configs let reviewers enter a numeric score within a defined range.

ax annotation-configs create \
  --name "Quality Score" \
  --space SPACE \
  --type continuous \
  --min-score 0 \
  --max-score 10 \
  --optimization-direction maximize

Create — Freeform

Freeform config