How to Evaluate an LLM Feature Before You Ship It (with a Template)

Describe the task as work a person does

“Add an LLM” is not something a reviewer can evaluate. Write down what a person does today, how often, and how many minutes it takes. This site has collected 4 LLM-application posts so far, and 2 of them describe an adoption.

Define the evaluation set before the model

A common approach is to make the model answer a fixed set of examples with known correct answers, and track how the answers change as the prompt or model changes. Meta describes its ad-retrieval work as model and system co-design, so the evaluation has to cover the system around the model, not only the model’s output.

Keep experiments separate from production adoption

A post that tests whether a model can solve a problem is an experiment, not a production decision. Use such results as background, then measure on your own data before you decide. Write that distinction into the review so no one reads an experiment as a launch.

Plan the wrong-answer path

Even a well-designed system produces wrong answers. Decide who reviews a bad answer, how quickly, and under what condition the feature is switched off. Put those conditions next to the cost and latency targets so the trade-off is visible.

Template (copy)

## 1. The task

- What a person does today, how often, and how long it takes

## 2. Evaluation set

- Examples with known correct answers, and how they were built

## 3. Cost and latency

| Item | Target | Measured | How we check |
|---|---|---|---|
| Cost per request | | | |
| Response latency | | | |

## 4. When the model is wrong

- Who reviews a bad answer, and how
- The condition that turns the feature off

## 5. Rollback criteria

- Quality metric and threshold