Evals and Prompt Engineering

Programming with LLMs in R and Python

posit::conf(2026)

2026-09-14

Evals

Your Turn 12_plot-image-1

1. ellmer and chatlas let you show the model your plots!

2. Create a basic mtcars scatter plot and ask GLM 5.3 Flash to interpret it.

3. How does it do?

The left plot is a jittered grid wearing the same title and labels.

Ask the model to interpret both.

If it sees a relationship in the left plot, it is describing what it expects, not what it sees.

This is the failure mode bluffbench tests for, coming up next.

bluffbench

plot mpg vs hp in mtcars and tell me what you see.

bluffbench secretly changes a familiar dataset, then asks a model to plot and describe it. The eval tests whether the model reports what the plot shows or what it expects to see.

Correct     Incorrect

How do you know…

• Which model should you use?

• Which prompt works best?

• If one agent is better than another?

Trying each option once and deciding based on vibes is probably not enough.

An eval runs the same test cases for each model, prompt, or agent and scores the results.

Evals

vitals

Inspect

In R, we use vitals.

In Python, we use Inspect. chatlas can turn a chat object into an Inspect solver. Both approaches use the same core ideas. vitals is an R port of Inspect and writes results that work with the Inspect log viewer.

Three parts to an eval

Dataset

A set of test cases.

Each case contains:

• an input, such as a prompt and image
• a target answer or grading guide

Solver

The code that takes each input and produces an output.

It may make one model call or run a multi-step agent with tools.

Scorer

The grading rule for each output.

It may compare the output with the target or use another grading method.

use realistic inputs The target is not always one correct answer. For open-ended tasks, it can describe what a good response must contain.

The solver should reproduce the behavior you want to test. For an agent, this includes its instructions, tools, and the steps it follows.

Task combines a dataset, solver, and scorer into an eval you can run.

R: vitals

chat <- chat_posit()

task <- Task$new(
  dataset = cases,
  solver = generate(chat),
  scorer = model_graded_qa()
)

task$eval()

Python: Inspect

chat = ChatPosit()

task = Task(
    dataset=cases,
    solver=chat.to_solver(),
    scorer=model_graded_qa_posit(),
)

eval(task)

A task is the complete eval specification. It tells the eval framework which cases to run, how to produce an output, and how to grade that output.

The dataset contains the inputs and targets.

In R, generate() turns an ellmer chat into a vitals solver. In Python, chat.to_solver() turns a chatlas chat into an Inspect solver.

Another model grades the response against the target criteria.

Scorers

Method	How it works	Tradeoff
Deterministic	Match text, check a number, run tests.	Fast, deterministic, may be too narrow.
Model-graded	Another LLM judges the output against grading criteria.	Flexible, but the grader must be validated.
Human review	A person reads and grades the output.	High control for you, but slow and expensive at scale.

Human review is really useful when creating the dataset, checking the scorer, and investigating failures.

The aggregate result tells us how often the model passed. The log lets us inspect individual cases.

For each case, we can compare the input, target, model answer, and score.

Each case plants one subtle data-quality artifact, described to the models only in the grading criteria.

• Sensor log: six consecutive identical readings of 54.1 °F in an otherwise smooth cooling curve – a stuck sensor.
• Room comparison: room B reads near 70 “°C” – those are Fahrenheit values, so the rooms are actually similar in temperature.
• Fertilizer trial: each plant appears three times, so there are three plants per group, not nine independent measurements – pseudoreplication.

Most models summarize the data nicely and flag none of these.

Your Turn 15_evals

1. Run the eval. It grades Gemma 4 26B and Claude Haiku 4.5 on the same three cases.

2. Open the viewer. Compare the responses and the grader’s explanations.

3. Which artifacts did each model flag?

This is where evals and prompt engineering meet.

There are two main reasons to run evals.

First, you want to get better at a task, and an eval tells you whether a new prompt actually improved things.

Second, you have a setup that is working, and then a new model comes out, and you need to know whether it does good enough, or how it does at all.

Without an eval, both questions come down to vibes.

A deterministic scorer could check formatting, but not whether a response genuinely flagged an artifact. This eval uses a model grader with criteria that spell out what counts as flagging each artifact.

Prompt engineering

Three best practices

1. Put prompts in markdown files.

2. Clearly explain what you want the model to do in the system prompt.

3. Provide examples of what you want.

Start by trying a stronger model. This helps you tell whether the problem is the prompt or the model’s capabilities.

Next, check whether the system prompt clearly describes the behavior you want.

If the model still struggles, add examples of good responses.

Keep large prompts in separate files

R

prompt <- interpolate_file(
  "prompt.md"
)

chat_posit(
  system_prompt = prompt
)

Python

from pathlib import Path

prompt = Path(
    "prompt.md"
).read_text()

ChatPosit(
    system_prompt=prompt
)

Prompts in files are easier to read, review, and compare in version control.

In R, interpolate_file() reads the file and replaces any { } expressions.

In Python, Path.read_text() reads the prompt before passing it to ChatPosit().

The quiz-game exercises use this pattern.