Checking AI-written health notes
Check whether links to a consultation transcript help reviewers catch mistakes in AI-written notes.
Technical
Bachelor’s · applied project
Final result
Review prototype and documented usability problems.
Material to create
20 scripted consultations and controlled draft notes
Write 20 fictional consultations (300–500 words), generate fixed-prompt Qwen2.5-7B-Instruct notes and clinically check/correct them. Allocate 5 notes each to clean, wrong-fact, omitted-action and both-error profiles. Preserve the original outputs and alteration key.
What to do
- Build a note-review screen with manually prepared links to the consultation.
How it is tested
Use the fictional consultation–note pairs; check link accuracy and observe 4 reviewers finding and correcting errors.
Before you start
Example tools · Qwen2.5-7B-Instruct, Python, Streamlit These are suggestions; equivalent tools are fine.
Access or people · Clinical review and professional reviewers required.
Research method · Build and test · 6 steps
- Read the starting sources and choose one established implementation method.
- Write the requirements, data fields, system diagram and test cases.
- Prepare 20 scripted consultations and controlled draft notes. Make the answer key and pass criteria before testing.
- Build a working version of review prototype and documented usability problems.
- Run function, integration and failure-case tests. Record each result.
- Run the practical evaluation and list the changes the system still needs.
Download the complete bachelor plan
Download bachelor instructionsMaster’s · research project
Final result
Note-review prototype, scripts and error-detection results.
Material to create
20 scripted consultations and controlled draft notes
Write 20 fictional consultations (300–500 words), generate fixed-prompt Qwen2.5-7B-Instruct notes and clinically check/correct them. Allocate 5 notes each to clean, wrong-fact, omitted-action and both-error profiles. Preserve the original outputs and alteration key.
What to do
- Pin model/tokenizer revision, Transformers version, quantization, prompt hash and deterministic decoding; preserve raw notes separately from corrected references.
- Represent note sentences and transcript character spans in JSONL, including an explicit omission layer; compare ordinary navigation with links plus an omission checklist.
- Verify link precision/coverage and key integrity; report TP/FP/FN by fault type, correction correctness, source openings and time, separating natural and planted faults.
How it is tested
8 health professionals review 10 non-overlapping notes each, with and without source links. Balance order and case difficulty. Keep fault counts hidden. Report missed faults, false alarms, correction accuracy and time; separate natural from planted errors.
Before you start
Example tools · Qwen2.5-7B-Instruct, Python, Streamlit These are suggestions; equivalent tools are fine.
Access or people · Clinical review and 8 professional reviewers required.
Start with these three actions
- Write 2 scripts and clinically review the facts and actions before generating notes. Confirm access to inference compute and record model, software and hardware versions.
- Generate and save both original and edited drafts; label natural errors separately from planted errors.
- Create two matched note sets and pilot the review instructions before recruiting clinicians.
Files, questions and sources
Pilot question: Do transcript links help reviewers correct faulty notes without wrongly flagging correct statements?
Literature search: clinical documentation AI note source attribution verification
Empty CSV templates: data-plan.csv · evaluation.csv · literature-matrix.csv
Download starter instructionsResearch method · Design, build and test (DSR) · 7 steps
- Read the newest papers and list the closest existing systems.
- Write down the versions, fields, data split, case assignment, random seeds and correct answers for 20 scripted consultations and controlled draft notes.
- Draw the user workflow, data model and system architecture. List the requirements and pass criteria.
- Build a working version of note-review prototype, scripts and error-detection results.
- Test every function, connection and failure case. Save the failed tests as well as the passed tests.
- Compare the system with the named alternative. Then run the user task or decision task in the assignment.
- Report the measured result, the failed cases and the design lessons another team can reuse.
Research question and sources
Research question: Test whether source links support verification or merely make AI notes look credible. Compare links on/off across clean, wrong-fact and omission profiles; distinguish false alarms, missed errors and verification effort.
Research result: A tested account of when traceability improves note-review decisions.
Current project literature
- Accuracy and Safety of AI-Enabled Scribe Technology: Instrument Validation Study (2025) peer-reviewed
Provides validated error categories and evidence of frequent errors in generated notes. - Using ChatGPT-4 to Create Structured Medical Notes From Audio Recordings of Physician-Patient Encounters: Comparative Study (2024) peer-reviewed
Reports omission, addition and incorrect-fact errors against consultation transcripts.
IS theory starting points
- Buçinca et al. (2021) — To Trust or to Think — Derive a competing explanation based on verification effort and overreliance.
- Vasconcelos et al. (2023) — Explanations Can Reduce Overreliance — Compare verification cost with trust as explanations of observed decisions.
Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points.
Search terms: clinical documentation AI note source attribution verification
Non-technical
Bachelor’s · applied project
Final result
A practical sign-off checklist supported by observations.
Material to create
10 scripted consultations
Create 10 clinically checked consultation–note pairs: 2 clean, 3 wrong-fact, 3 omitted-action and 2 both-error notes. Keep original generation, corrected baseline and planted alteration keys separate; hide fault counts from reviewers.
What to do
- Document how staff check a draft health note before sign-off.
How it is tested
Use paper consultation–note pairs in 4 think-aloud sessions; classify missed facts, omissions and checking steps.
Before you start
Example tools · LibreOffice Writer/Calc, audio recorder with consent; no programming required. These are suggestions; equivalent tools are fine.
Access or people · Recruit health professionals; clinically review fictional material.
Research method · Study and improve · 6 steps
- Read the starting sources and write the exact information problem.
- Prepare 10 scripted consultations and a separate answer sheet or coding sheet.
- Run one pilot and fix unclear questions.
- Collect the named evidence with consent.
- Group the findings with the stated categories and check the answer sheet.
- Produce a practical sign-off checklist supported by observations. List the three most useful changes.
Download the complete bachelor plan
Download bachelor instructionsMaster’s · research project
Final result
Error table and recommendations for reviewing AI notes.
Material to create
10 scripted consultations + 8 reviewer sessions
Create 10 clinically checked consultation–note pairs: 2 clean, 3 wrong-fact, 3 omitted-action and 2 both-error notes. Keep original generation, corrected baseline and planted alteration keys separate; hide fault counts from reviewers.
What to do
- Prepare note/transcript pairs as documents; no application development.
- Observe reviewers checking facts, then interview them about approval decisions.
How it is tested
Think-aloud document review; count errors found and code checking strategies and reasons for approval.
Before you start
Example tools · LibreOffice Writer/Calc, audio recorder with consent; no programming required. These are suggestions; equivalent tools are fine.
Access or people · Recruit health professionals; clinically review fictional material.
Start with these three actions
- Prepare a pilot with 2 examples from: 10 scripted consultations + 8 reviewer sessions. Write the task questions and a reference answer sheet.
- Write a recruitment message, information sheet and consent form for the participants named above. Agree privacy handling with the supervisor before contact.
- Pilot one session after approval; revise unclear questions, finalise the task sets and coding categories, then recruit the planned sample.
Files, questions and sources
Pilot question: How do health professionals decide that an AI-drafted note is ready to approve?
Literature search: clinical documentation AI note source attribution verification qualitative scenario study
Interview prompts
- Which statement would you correct or verify?
- What transcript passage supports your decision?
- Would you approve this note, and what remains unchecked?
Empty CSV templates: study-plan.csv · coding-sheet.csv · literature-matrix.csv
Download starter instructionsResearch method · Study, compare and explain · 7 steps
- Read the newest papers and write one exact research question.
- State which people, cases or documents you will study and what you will compare.
- Write the selection rules, questions and analysis steps for 10 scripted consultations + 8 reviewer sessions.
- Run one pilot. Fix unclear questions or categories, then keep the guide unchanged.
- Collect the named interviews, cases or documents with consent.
- Analyse them with the stated comparison or coding method. Keep disagreements and missing data.
- Report the answer, the evidence and the practical output named in the assignment.
Research question and sources
Research question: Explain why reviewers accept some unsupported statements but challenge others. Analyse checking effort, perceived system competence and sign-off responsibility; seek cases that contradict the initial explanation.
Research result: A theory-based model of verification and acceptance, with specific design implications.
Current project literature
- Accuracy and Safety of AI-Enabled Scribe Technology: Instrument Validation Study (2025) peer-reviewed
Provides validated error categories and evidence of frequent errors in generated notes. - Using ChatGPT-4 to Create Structured Medical Notes From Audio Recordings of Physician-Patient Encounters: Comparative Study (2024) peer-reviewed
Reports omission, addition and incorrect-fact errors against consultation transcripts.
IS theory starting points
- Vasconcelos et al. (2023) — Explanations Can Reduce Overreliance — Compare verification cost with trust as explanations of observed decisions.
- Orlikowski & Gash (1994) — Technological Frames — Compare how roles interpret the purpose, operation and use of the same system.
Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points.
Search terms: clinical documentation AI note source attribution verification qualitative scenario study
Your degree, track and shortlist are saved in this browser.