# OfferKu AI Product Evaluation CSV Template

Version: 2026-08-13
Scope: offline evaluation of generative AI features, RAG answers, tool-using agents, and workflow agents.

`offerku-ai-eval-template.csv` contains 12 entirely synthetic examples with no personally identifiable information. It is a reusable starting point, not a model benchmark result and not a representation of production data.

## Quick start

1. Copy the CSV and freeze an immutable version for the evaluation run.
2. Keep each `case_id` stable and add cases for the task's actual risk tiers and important slices.
3. Run the baseline and candidate with the same inputs, context, prompt, and tool constraints.
4. Store the raw responses in `baseline_output` and `candidate_output`.
5. Score each rubric, then have a human reviewer fill `human_label`, `failure_type`, and `reviewer_notes`.
6. Check critical failures first, then aggregate separately by task, risk tier, and slice instead of relying only on a global average.

A simple 0, 1, 2 scale works well: 0 means unmet, 1 means partially met, and 2 means met. A case that matches `critical_failure` should be tracked separately and should normally block release; strong average scores must not cancel it out.

## Column dictionary

| Column | Meaning |
| --- | --- |
| `case_id` | Stable unique case identifier; create a new ID when the case meaning changes. |
| `task_type` | Task family such as knowledge answer, summary, tool call, or external action. |
| `risk_tier` | Risk level; this template uses `low`, `medium`, `high`, and `critical`. |
| `slice` | Diagnostic subgroup such as language, missing evidence, or tenant isolation. |
| `input` | User input sent to the system. |
| `context` | Available facts, retrieved material, permissions, and tool constraints. |
| `expected_behavior` | Observable behavior that defines a successful response. |
| `critical_failure` | Failure that requires priority review or release blocking. |
| `rubric_faithfulness` | Scoring instruction for grounding and factual faithfulness. |
| `rubric_task_completion` | Scoring instruction for task completion. |
| `rubric_boundary_handling` | Scoring instruction for authorization, safety, privacy, and uncertainty boundaries. |
| `baseline_output` | Raw output from the current baseline; intentionally blank in the template. |
| `candidate_output` | Raw output from the candidate; intentionally blank in the template. |
| `human_label` | Human result such as `pass`, `fail`, or `needs_review`. |
| `failure_type` | Failure taxonomy label; it can remain blank for passing cases. |
| `reviewer_notes` | Evidence, disagreements, and audit notes. |

## Evaluation workflow

- Define release gates first. Set an explicit critical-failure threshold and independent gates for high-risk slices.
- Maintain both a development set and an independent holdout set. Iterate on the development set and use the holdout for release decisions to reduce overfitting.
- Record the model version, prompt version, retrieval-index version, tool definitions, parameters, random seed, and run time.
- Automated judges can expand coverage, but high-risk cases, judge disagreements, boundary cases, and sampled results need human review.
- Compare both aggregate and slice-level results, and retain per-case evidence so regressions can be diagnosed.
- Turn observed failure modes into new cases. Do not remove a failed case merely because the prompt was changed.

## Privacy and CSV safety

- Every organization, product, order, account, and asset in this template is fictional. Prefer synthetic or approved de-identified data when extending it. Do not include names, contact details, secrets, access tokens, raw customer messages, or regulated data.
- Spreadsheet software may interpret cells beginning with `=`, `+`, `-`, or `@` as formulas. This template contains none. Validate or escape dangerous prefixes before importing third-party output, and use the spreadsheet application's text-import mode where possible.
- Do not concatenate model output into CSV manually. Use an RFC 4180-compatible CSV library, escape double quotes, and retain an immutable copy of each raw output.
- Version the dataset, rubrics, and run configuration together. Keep a dated changelog with the reason for every case addition or removal so historical baselines are not silently changed.

## Human review guidance

At minimum, manually review every `critical` case, every critical failure, automated-judge disagreement, output containing new factual claims, and attempted external action. Reviewers should judge only against the current case's `input`, `context`, expected behavior, and rubrics, then leave traceable evidence in `reviewer_notes`.

## Official references

- [OpenAI Evals guide](https://developers.openai.com/api/docs/guides/evals)
- [OpenAI Evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices)
- [Google Cloud Vertex AI Agent evaluation](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/evaluate)
- [Microsoft Foundry Agent evaluators](https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/agent-evaluators)
- [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework)

Together, these sources support task-driven cases, clear grading criteria, slice-level analysis, combined automated and human review, and risk-prioritized evaluation. The right metrics and release gates still depend on the product's actual risks and use case.
