> ## Documentation Index
> Fetch the complete documentation index at: https://docs.intellikon.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Datasets and evaluations
> Collect sample inputs, run prompts against them, score the answers and evaluate every new version automatically.
An evaluation runs a prompt against sample inputs and, if you add scorers, scores each answer, so you can see whether a new version or a different model does better before you release it. The inputs come from a dataset or from values you type in. The scores come from scorers: marking guides that an AI judge applies to each answer. Nucleus includes **Coherence**, **Faithfulness** and **Relevance** scorers, and you can write your own.
## Before you start
Tenant Owners, Workspace Admins and Workspace Members can create datasets and run evaluations. Only Tenant Owners and Workspace Admins can create or change scorers. The models you can compare are the ones your workspace allows for Nucleus in Studio settings, under **AI Models**.
## Start a dataset
A dataset is a named set of input rows, each holding a value for the prompt's variables and, if you want, the answer you expect.
Select **Datasets**, then **New dataset**.
Choose **From Observability** to use real requests, **Manual** to type rows yourself, or **Upload CSV / JSONL** to use a file.
Enter a **Name**. The **Slug** fills in from it.
Add a **Description** if you want.
Add **Tags** if you want.
Under **Rows**, the form shows the fields for the source you chose. Next, [add the rows](#add-the-rows).
## Add the rows
If you chose **From Observability**, choose the **Prompt**.
If you chose **From Observability**, choose how many days back to look under **Window (days)**.
If you chose **Manual**, fill in each row.
If you chose **Manual** and need more rows, select **Add row**.
If you chose **Upload CSV / JSONL**, drop the file in.
If you added a file, select **Preview** to check it.
Select **Create dataset**.
For **Manual** rows, write the inputs under **input\_variables (JSON)** with one entry per variable, such as `{ "ticket": "The van will not start" }`. Add the answer you hope for under **expected\_output (optional)**.
The dataset opens, and it is listed in **Datasets** with its number of rows. Next, [set up an evaluation](#set-up-an-evaluation) with it.
## Set up an evaluation
Select **Evaluations**, then **New Evaluation**.
Choose **Model comparison** to try one version on several models, **Version comparison** to try several versions on one model, or **Full matrix** to try every combination.
Pick the **Prompt**.
Pick the versions to compare.
Pick the models to compare.
Under **Scorers**, select the scorers to mark each answer with. This is optional.
**Input** is next on the form. Next, [add input and run the evaluation](#add-input-and-run-the-evaluation).
## Add input and run the evaluation
Under **Input**, stay on **Manual** or choose **From dataset**.
If you stayed on **Manual**, fill in the values.
If you chose **From dataset**, pick the **Dataset**.
If you chose **From dataset**, pick its **Version**.
Change the name if you want.
Select **Run evaluation**.
**Results** shows the run's progress, then the answers once it finishes. The run is also listed under **Runs**. Every answer is a real request to the model, and each scorer adds a request to its judge. Next, [read the results](#read-the-results).
## Read the results
Select a run under **Runs** to open it. Each result shows the model and version, the answer, its scores and how long it took. **Mean per scorer** at the bottom gives the average score for each scorer across the run.
To record which result you prefer, select the thumbs-up on it. The result is marked with a crown, and the run's **Winner** column in **Runs** shows it. Next, [evaluate every new version automatically](#evaluate-every-new-version-automatically).
## Create a scorer
Select **Evaluations**, then the **Scorers** tab, then **New scorer**.
Enter a **Name**.
Enter a **Slug**.
Enter a **Description** that helps your team pick it.
Under **Rubric**, describe what each score means. The judge sees exactly this, so be specific.
Pick the **Judge model**.
Pick the **Scale**.
Select **Create scorer**.
The scorer appears under **Workspace scorers**, and you can add it to any evaluation in this workspace. Next, [set up an evaluation](#set-up-an-evaluation) with it.
## Evaluate every new version automatically
With automatic evaluation on, every version you save is run against a dataset and scored, and Nucleus uses the result when you promote.
Select **Prompts**, then the prompt.
Select **Advanced configuration**.
Tick **Auto-evaluate on save**.
Pick a **Default dataset**.
Select one or more **Default scorers**.
Under **Pass threshold (optional)**, enter the average score a version needs.
Select **Save auto-eval settings**.
From now on, each version you save is evaluated. Next, [promote the version](/products/nucleus/releasing-prompts).
## Troubleshooting
Choose a prompt, and as many models and versions as the evaluation type asks for. **Model comparison** needs at least two models, and **Version comparison** at least two versions.
Check the row's braces and quotation marks. Each entry needs its name and its value in double quotes, such as `{ "ticket": "The van will not start" }`.
Use only lower-case letters, numbers and hyphens in the **Slug**, such as `van-breakdown-tickets`.
Automatic evaluation needs a **Default dataset** and at least one **Default scorer**. Pick both, then select **Save auto-eval settings**.
Only Tenant Owners and Workspace Admins can create scorers. You can still use the ones listed in any evaluation.