Understanding public-service guidance Personal proposal by Per-Arne Andersen D3 · Spring 2027 https://thesis.uya.no/proposals/public-services-made-clear/ Help readers find the correct passage in NAV’s financial-assistance guidance. TECHNICAL - Save a dated copy of NAV’s “Økonomisk sosialhjelp” page. - Build paragraph search with BM25 ranking, source links and a “no matching passage” response. DATA (mixed) NAV guidance + 40 labelled questions Freeze expanded sections and paragraph IDs. Write 30 answerable/10 out-of-scope questions. Split by intent: development 7/3; held-out 23/7. Independently check acceptable passages. NAV: Økonomisk sosialhjelp: https://www.nav.no/okonomisk-sosialhjelp METHOD BM25 versus term-overlap search, with identical processing and interface. Score held-out hit@3, coverage and incorrect refusals separately. Twelve Norwegian-reading adults complete 8 disjoint tasks, 4 per mode; measure comprehension, source checking and unsupported reliance. OUTPUT Search prototype, question set and retrieval comparison. BACHELOR Task: Build a BM25 paragraph search over the specified NAV page. Data: NAV guidance + 40 labelled questions (mixed) Freeze expanded sections and paragraph IDs. Write 30 answerable/10 out-of-scope questions. Split by intent: development 7/3; held-out 23/7. Independently check acceptable passages. Requires: Recruit Norwegian-reading adults and an independent key reviewer; NAV staff access is unnecessary. Method: Use the frozen page and labelled questions; check retrieval hits and observe adults finding the correct source paragraph. Plan 4 participant sessions. Output: Search prototype, test questions and usability findings. Use relevant literature to justify the established approach; a new research contribution is not the aim of this proposal. MASTER Investigate retrieval failure and abstention for Norwegian public-service questions. Compare term overlap and BM25 on answerable/unanswerable intents; study how source presentation affects successful information finding without inventing answers. Intended contribution: A bounded evaluation of retrieval and evidence-use failures, with design implications. STARTING PAPERS Gao et al. (2024 revision) — Retrieval-Augmented Generation: A Survey: https://arxiv.org/abs/2312.10997 Separate retrieval, evidence use and output evaluation; trace newer retrieval work. Vessey & Galletta (1991) — Cognitive Fit: https://pubsonline.informs.org/doi/10.1287/isre.2.1.63 Test whether a representation helps one kind of task more than another. Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points. START HERE Tools: Python, BM25 library, Streamlit 1. Save a dated NAV page snapshot and split it into numbered paragraphs. 2. Write 5 answerable and 2 out-of-scope questions; ask a second reader to label the matching passages. 3. Freeze the question split, build keyword search, then add BM25 without consulting held-out answers. Literature search: public service information retrieval citizen comprehension Study controls: - Keep paraphrases of an intent in one split. Use development questions only for each abstention threshold. - Score hit@3 on answerable questions before abstention; separately report unsupported accepted results and unnecessary refusals. - Retrieve passages only: no generated advice or benefit eligibility decisions. - Before collecting participant data, agree consent, storage and withdrawal handling with the supervisor. Use participant codes, not names, in study files. - For the master’s study, use the research task above to define the factors and comparisons in this pilot plan. Preregister one primary outcome and feasible scope after the literature review; do not add every possible model or interface variant. REQUIRES Recruit 12 Norwegian-reading adults and an independent key reviewer; NAV staff access is unnecessary. -------------------- NON-TECHNICAL - Use the existing NAV page and fictional information-seeking questions. - Observe where readers look and ask them to explain the answer in their own words. DATA (mixed) NAV guidance + 8 task cards Create tasks about required documents, application steps and finding human contact. Recruit 12 adult readers; collect no benefit applications. NAV: Økonomisk sosialhjelp: https://www.nav.no/okonomisk-sosialhjelp METHOD Think-aloud sessions; score passage finding and comprehension against a reviewed answer sheet. Code navigation and wording problems. OUTPUT Eight task results and a prioritised list of page improvements. BACHELOR Task: Identify obstacles when users look for information on the NAV page. Data: NAV guidance + 8 task cards (mixed) Create tasks about required documents, application steps and finding human contact. Requires: Recruit readers and obtain an independent answer-sheet review. Method: Run task walkthroughs on the existing page; group failures by wording, navigation and missing context. Plan 4 participant sessions. Output: A ranked list of concrete page changes. Use relevant literature to justify the established approach; a new research contribution is not the aim of this proposal. MASTER Explain why some users cannot turn located information into an answer. Separate navigation, terminology and task-context failures; use cognitive fit and information quality to explain contrasting cases. Intended contribution: A model of information-use barriers supported by task evidence. STARTING PAPERS Vessey & Galletta (1991) — Cognitive Fit: https://pubsonline.informs.org/doi/10.1287/isre.2.1.63 Test whether a representation helps one kind of task more than another. Wang & Strong (1996) — Beyond Accuracy: https://www.tandfonline.com/doi/abs/10.1080/07421222.1996.11518099 Distinguish accuracy, completeness, representation and fitness for the task. Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points. START HERE Tools: LibreOffice Writer/Calc, audio recorder with consent; no programming required. 1. Prepare a pilot with 2 examples from: NAV guidance + 8 task cards. Write the task questions and a reference answer sheet. 2. Write a recruitment message, information sheet and consent form for the participants named above. Agree privacy handling with the supervisor before contact. 3. Pilot one session after approval; revise unclear questions, freeze the task sets and coding categories, then recruit the planned sample. Literature search: public service information retrieval citizen comprehension qualitative scenario study Study controls: - Before collecting participant data, agree consent, storage and withdrawal handling with the supervisor. Use participant codes, not names, in study files. - Pilot separately, then freeze the questions and coding plan. Check objective answer keys independently; keep an audit trail of coding, including disagreements. - For comparisons, counterbalance order and case assignment; do not show a person both versions of one case. Report participant-level findings, not repeated tasks as independent people. REQUIRES Recruit readers and obtain an independent answer-sheet review. Bachelor: apply established methods and evaluate a practical solution or study. Master: position a research question in current scientific literature, investigate a mechanism or unresolved problem, and explain the contribution. Final scope is agreed with me. Study sizes are proposed; participant recruitment and planned materials are not already arranged.