# Repeat the Dayora fictional-journal benchmark

Version 2.0.0. Published by Dayora. The fixture is CC0. All people and events are fictional. No member data is included.

## Download and preserve the inputs

Download [the 120-entry fixture](https://www.dayora.ai/research/journal-benchmark-v2.json) and [the full recorded run](https://www.dayora.ai/research/journal-benchmark-v2-results.json). Compare the fixture's SHA-256 with `fixtureSha256` in the run before using it. Preserve the original English and entry dates. `role` and `evidenceGroups` are scoring annotations, not context to pass to the answer model.

This is a benchmark JSON format, not a promised import format for Dayora or another app. Use the tested product's supported entry-creation method. If it cannot preserve entry dates, disclose that difference. Use a fresh test account or isolated local database and never use a real person's journal.

## Run the same questions

1. Load all 120 entries, preserving their dates, and allow normal indexing to finish. Record any skipped entries and do not silently test a smaller favorable subset.
2. Record product, plan, model if known, version, date, access restrictions and whether the run uses the product UI or a local pipeline.
3. Ask each of the 12 questions three times in fresh conversations. Retain every answer, refusal and error. Do not retry failed generations to obtain a preferred score.
4. Save the passages actually returned by retrieval if the product exposes them. Do not insert the expected evidence into the answer context. If retrieval is not exposed, mark retrieval unmeasured rather than infer it from a good answer.
5. Preserve raw answers in the original English. Review them against every criterion in that question's frozen checklist, using the full fixture to check identities and corrections.

## Score two different things

**Retrieval:** Each `evidenceGroups` array requires at least one of its listed entries. Complete evidence retrieval means every required group has a returned entry. There are 10 answerable questions and three attempts each: 30 eligible attempts. The two missing-fact questions have no positive evidence target and must not be counted as retrieval successes. No answer model output can repair a missing retrieval record.

**Answer completeness:** Pass only when every checklist item is met. Preserve an item-level reason. A specific required detail or distinction that is omitted fails that item, even if the core answer is correct. A `nothing_found` outcome fails an answerable question but can satisfy an absent-fact criterion. Record errors separately and retain all 36 scheduled attempts in the overall denominator. Also report pipeline completion so errors cannot disappear behind a percentage.

These are checklist outcomes, not a score for all factual claims, therapeutic quality, warmth or citation precision. The current Dayora answer prompt discourages weekday mentions; that conflicts with the bicycle question's explicit Sunday detail. The published run leaves that mismatch visible rather than changing the rubric after seeing the outputs.

## What the published Dayora run actually used

The run used unmodified production indexing, retrieval and answer functions against a local database. It indexed all 120 fictional entries at once, selected at most 14 passages from 60 vector candidates, and generated each answer afresh. Two independent question jobs ran at a time. The record provides execution timestamps, models, pipeline commit, code hashes, fixture hash and per-attempt latency. Database entry IDs were normalized to fixture IDs for readability; answer text was preserved.

Dayora's service source repository is private. The public kit provides inputs, protocol, criteria, retrieved evidence and outputs for replication or scrutiny; it is not a standalone distribution of the service. Researchers with repository access can run `scripts/run-journal-benchmark-v2.ts` using the project's local development environment. The script refuses a non-local database, creates one fictional account and removes it after the run. Other researchers can repeat the protocol through an app they can access, recording any differences.

Production Ask Your Journal requires Plus and rollout access. This local run does not test UI behavior, billing or entitlements. Other apps have not been assigned a score.

## Limits and interpretation

One deliberately constructed English diary with repeated distractor themes is not a representative user sample. The entries were loaded at once, so this is not elapsed long-term memory. Three repeats give a limited view of variation. Dates affect retrieval. No clinical, account-deletion or security conclusions can be drawn.

Publisher review used Codex assistance and is not independent adjudication. Inspect the individual reasons, repeat the test and publish disagreements. The current pipeline attaches every retrieved entry ID to an answer, so citation presence alone is not a precision score.