Skip to main content
An evaluation runs a prompt against sample inputs and, if you add scorers, scores each answer, so you can see whether a new version or a different model does better before you release it. The inputs come from a dataset or from values you type in. The scores come from scorers: marking guides that an AI judge applies to each answer. Nucleus includes Coherence, Faithfulness and Relevance scorers, and you can write your own.

​
Before you start

Tenant Owners, Workspace Admins and Workspace Members can create datasets and run evaluations. Only Tenant Owners and Workspace Admins can create or change scorers. The models you can compare are the ones your workspace allows for Nucleus in Studio settings, under AI Models.

​
Start a dataset

The top of the New dataset form, with the three sources, From Observability chosen, and Name & Tags.
A dataset is a named set of input rows, each holding a value for the prompt’s variables and, if you want, the answer you expect.
1

Start a dataset

Select Datasets, then New dataset.
2

Choose where the rows come from

Choose From Observability to use real requests, Manual to type rows yourself, or Upload CSV / JSONL to use a file.
3

Name it

Enter a Name. The Slug fills in from it.
4

Describe it

Add a Description if you want.
5

Tag it

Add Tags if you want.
Under Rows, the form shows the fields for the source you chose. Next, add the rows.

​
Add the rows

1

Choose the prompt

If you chose From Observability, choose the Prompt.
2

Choose how far back to look

If you chose From Observability, choose how many days back to look under Window (days).
3

Fill in the rows

If you chose Manual, fill in each row.
4

Add more rows

If you chose Manual and need more rows, select Add row.
5

Add the file

If you chose Upload CSV / JSONL, drop the file in.
6

Check the file

If you added a file, select Preview to check it.
7

Create it

Select Create dataset.
For Manual rows, write the inputs under input_variables (JSON) with one entry per variable, such as { "ticket": "The van will not start" }. Add the answer you hope for under expected_output (optional). The dataset opens, and it is listed in Datasets with its number of rows. Next, set up an evaluation with it.

​
Set up an evaluation

The New Evaluation tab, with the three evaluation types to choose from, then Configure, Scorers and Input, and the name box and Run evaluation button at the bottom.
1

Start an evaluation

Select Evaluations, then New Evaluation.
2

Choose what to compare

Choose Model comparison to try one version on several models, Version comparison to try several versions on one model, or Full matrix to try every combination.
3

Choose the prompt

Pick the Prompt.
4

Choose the versions

Pick the versions to compare.
5

Choose the models

Pick the models to compare.
6

Add scorers

Under Scorers, select the scorers to mark each answer with. This is optional.
Input is next on the form. Next, add input and run the evaluation.

​
Add input and run the evaluation

1

Choose the input

Under Input, stay on Manual or choose From dataset.
2

Fill in the values

If you stayed on Manual, fill in the values.
3

Pick the dataset

If you chose From dataset, pick the Dataset.
4

Pick the dataset version

If you chose From dataset, pick its Version.
5

Name it

Change the name if you want.
6

Run it

Select Run evaluation.
Results shows the run’s progress, then the answers once it finishes. The run is also listed under Runs. Every answer is a real request to the model, and each scorer adds a request to its judge. Next, read the results.

​
Read the results

Select a run under Runs to open it. Each result shows the model and version, the answer, its scores and how long it took. Mean per scorer at the bottom gives the average score for each scorer across the run. To record which result you prefer, select the thumbs-up on it. The result is marked with a crown, and the run’s Winner column in Runs shows it. Next, evaluate every new version automatically.

​
Create a scorer

The New scorer window, with Slug, Name, Description and the Rubric.
1

Open the scorers

Select Evaluations, then the Scorers tab, then New scorer.
2

Name it

Enter a Name.
3

Give it a slug

Enter a Slug.
4

Describe it

Enter a Description that helps your team pick it.
5

Write the rubric

Under Rubric, describe what each score means. The judge sees exactly this, so be specific.
6

Choose the judge

Pick the Judge model.
7

Choose the scale

Pick the Scale.
8

Create it

Select Create scorer.
The scorer appears under Workspace scorers, and you can add it to any evaluation in this workspace. Next, set up an evaluation with it.

​
Evaluate every new version automatically

Advanced configuration open on a prompt, with the Auto-evaluation switch, the Auto-evaluate on save box, the Default dataset menu, the Pass threshold field, the Default scorers and Save auto-eval settings.
With automatic evaluation on, every version you save is run against a dataset and scored, and Nucleus uses the result when you promote.
1

Open the prompt

Select Prompts, then the prompt.
2

Open the settings

Select Advanced configuration.
3

Turn it on

Tick Auto-evaluate on save.
4

Choose the dataset

Pick a Default dataset.
5

Choose the scorers

Select one or more Default scorers.
6

Set a pass mark if you want one

Under Pass threshold (optional), enter the average score a version needs.
7

Save

Select Save auto-eval settings.
From now on, each version you save is evaluated. Next, promote the version.

​
Troubleshooting

Choose a prompt, and as many models and versions as the evaluation type asks for. Model comparison needs at least two models, and Version comparison at least two versions.
Check the row’s braces and quotation marks. Each entry needs its name and its value in double quotes, such as { "ticket": "The van will not start" }.
Use only lower-case letters, numbers and hyphens in the Slug, such as van-breakdown-tickets.
Automatic evaluation needs a Default dataset and at least one Default scorer. Pick both, then select Save auto-eval settings.
Only Tenant Owners and Workspace Admins can create scorers. You can still use the ones listed in any evaluation.