Why hallucinated action items are harder to notice than transcript errors
AI-generated meeting notes usually earn trust because the transcript looks plausible. The bigger risk is subtler: a summary can sound correct while inventing a commitment, a next step, or an owner. That’s different from a misheard word. A hallucinated action item often fits the meeting’s “shape” (people discussed a topic; therefore an action exists) even when no one actually agreed to do anything.
Teams feel this most when notes are auto-synced into task tools and CRMs. A single invented follow-up can create quiet downstream work, misreport pipeline status, or trigger unnecessary escalations. This is why QA for meeting summaries needs to focus on verifying commitments, not just checking whether the meeting was captured.
The statistical spot-check idea and why it works
Full review of every meeting summary doesn’t scale. A practical alternative is a statistical spot-check: you review a small, randomly selected subset of action items and treat that as a quality signal for the whole stream of notes. If the spot-check finds a non-trivial hallucination rate, you tighten controls (more review, stricter templates, different model settings, or better guardrails). If it stays low and stable, you can keep automation while still monitoring risk.
This approach borrows from classic quality assurance: you don’t inspect every unit, you audit enough units to detect when a process drifts. The key is to define what you’re sampling (action items, not whole summaries) and what counts as a defect (anything that isn’t supported by the transcript).
Define what “hallucinated” means in meeting notes
Before sampling, align on defect definitions. A useful definition for a hallucinated action item is: a claimed next step, owner, or deadline that cannot be directly supported by the transcript. That includes several common patterns:
- Invented commitment: “Alex will send the deck” when no one volunteered or was assigned.
- Owner swap: the action exists, but the note assigns it to the wrong person.
- Deadline invention: “by Friday” appears even though timing wasn’t agreed.
- Overconfident interpretation: a tentative idea (“maybe we should…”) becomes a firm task.
- Scope expansion: turning a discussion topic into a multi-step plan that never happened.
It’s helpful to distinguish these from ordinary summarization judgment calls. Condensing “we should look into it” into “investigate options” may be acceptable if it remains clearly tentative. The QA goal is to prevent hard commitments from being fabricated.
Set up the spot-check workflow
1) Choose the sampling unit and the population
Sample at the action-item level, not at the meeting level. One meeting may have zero action items; another may have ten. Your population is the total set of action items produced by your note generator over a time window (for example, one week).
If your tool stores structured action items separately from the narrative summary, use that. Products like Fathom already separate transcript, summary, and action items, which makes sampling easier because you’re not guessing which sentences are meant to be tasks.
2) Randomize selection to avoid “easy” audits
Spot-checks fail when reviewers subconsciously pick the meetings they remember or the customers they care about. Use simple randomization:
- Export a list of all action items created in the period.
- Assign each an ID.
- Use a random number generator to pick n items to review.
Keep the method lightweight. The goal is consistent monitoring, not perfect sampling theory.
3) Decide the minimum evidence requirement
For each sampled action item, require a “transcript anchor”: a direct supporting moment in the transcript that shows agreement on the action. You don’t need verbatim quotes in the final notes, but you do need a reviewer to locate the supporting exchange quickly.
If the reviewer cannot find the anchor in a reasonable time (for example, within 60–90 seconds), treat it as unsupported. In practice, “can’t find it quickly” correlates strongly with “it isn’t there,” and it keeps the audit economical.
How to score each action item
Use a simple rubric so results are comparable across reviewers:
- Verified: transcript clearly supports the task, owner, and timing (if timing is stated).
- Partially supported: the task exists, but owner or due date is wrong/overstated.
- Unsupported (hallucinated): no transcript evidence for the commitment.
- Ambiguous: discussion suggests a possible next step, but no clear assignment. (Treat as a process smell; many teams count this as a defect if it was stated as firm.)
Track defect type as well as defect count. “Deadline invention” often points to prompt/template issues, while “owner swap” may point to speaker identification or attribution problems.
From sample results to an actionable quality signal
After reviewing n action items, compute:
- Hallucination rate: unsupported / n
- Commitment integrity rate: verified / n
- Attribution error rate: (owner swap + deadline invention) / n
You don’t need complex statistics to benefit. What matters is trend detection: if the hallucination rate rises week-over-week, something changed (meeting types, vocabulary, model settings, integration behavior, or user expectations).
To keep the process operational, set a simple policy threshold. Example: if more than 2 out of 30 sampled action items are unsupported, require a human review step before pushing tasks into downstream systems for the following week. If it’s 0 out of 30 for several weeks, you can loosen controls.
Where spot-check QA fits into modern meeting-note stacks
Spot-checking is most valuable when notes are used as system inputs. If your summaries flow into Asana, Notion, Salesforce, or Slack, an invented action item becomes “real” very quickly. The QA process should be aligned with integration pathways: the stricter the automation, the stricter the sampling policy.
It also pairs well with operational monitoring. If you already run internal automations, treat hallucination rate like any other workflow metric. The same mindset used for detecting stuck DAG steps in internal automations applies here: define a failure mode, sample it, and respond before it cascades.
Reducing hallucinations without slowing teams down
Spot-checking tells you that you have a problem; you still need levers to fix it. The most effective interventions are usually procedural rather than “more AI”:
- Use a commitment template: require action items to be phrased as “Owner + verb + object (+ due date if stated).”
- Separate decisions from tasks: prevent “we decided X” from being reframed as “do Y.”
- Gate task creation: allow auto-created tasks only when the action item has an explicit owner and a transcript anchor, otherwise keep it as “candidate follow-up.”
- Improve vocabulary and names: custom vocabularies and accurate speaker labeling reduce owner swaps.
Tools like Fathom are helpful in this phase because you can quickly jump from summary to transcript, search across meetings, and verify whether an action item reflects an actual agreement—without making the reviewer rewatch a full recording.
