Reviewing AI-generated code
Personal proposal by Per-Arne Andersen
S2 · Spring 2027
https://thesis.uya.no/proposals/review-with-reasons/

Measure whether test evidence helps developers reject faulty AI-generated Python functions.

TECHNICAL
- Generate one greedy Qwen2.5-Coder-7B-Instruct completion for each HumanEval task.
- Build a review screen showing requirement, code and test failures.

DATA (mixed)
HumanEval/0–39 + Qwen Coder completions
Freeze model revision and prompt; save one greedy completion per task. Prepare independent requirement-based audit tests before generation; retain failed outputs.
HumanEval: https://github.com/openai/human-eval
Qwen2.5-Coder-7B-Instruct: https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct
Transformers generation settings: https://huggingface.co/docs/transformers/main_classes/text_generation

METHOD
Report the 40-completion published-test pass fraction separately. Eight developers review 10 disjoint cases with/without test evidence. Score false acceptance/rejection against independently verified faults, plus time and requests for more checking.

OUTPUT
Review screen, completions and acceptance-error results.

BACHELOR
Task: Build a code-review screen with links to tests and source lines.
Data: HumanEval/0–39 + Qwen Coder completions (mixed)
Freeze model revision and prompt; save one greedy completion per task. Prepare independent requirement-based audit tests before generation; retain failed outputs.
Requires: Inference compute, developers and independent correctness review; no unseen-task or general model-performance claim.
Method: Use the fixed HumanEval completions and independent audit tests; observe reviewers detecting and explaining faults. Plan 4 participant sessions.
Output: A review prototype and concrete usability fixes.
Use relevant literature to justify the established approach; a new research contribution is not the aim of this proposal.

MASTER
Investigate whether evidence links improve detection of AI-code faults or reinforce misleading tests. Use independent audit tests and unchanged generated outputs; analyse natural error types and reviewer verification behaviour separately.
Intended contribution: An evaluation linking test adequacy, traceability and human review effectiveness.

STARTING PAPERS
Liu et al. (2023) — Is Your Code Generated by ChatGPT Really Correct?: https://arxiv.org/abs/2305.01210
Distinguish passing a public test suite from independently assessed correctness.
Vasconcelos et al. (2023) — Explanations Can Reduce Overreliance: https://arxiv.org/abs/2212.06823
Compare verification cost with trust as explanations of observed decisions.

Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points.

START HERE
Tools: Qwen2.5-Coder-7B-Instruct, Python, HumanEval, Streamlit
1. Download HumanEval and the named model; record revisions, prompt and decoding settings. Confirm access to inference compute and record model, software and hardware versions.
2. Generate tasks 0–2 and run the published tests in an isolated environment with no credentials.
3. Prepare the same code-review layout in both conditions, differing only in visible test evidence.
Literature search: AI generated code review trust test evidence
Study controls:
- Use do_sample=False and num_beams=1, not temperature 0 alone. Pin tokenizer, runtime, quantisation, prompts and output limit.
- Published tests alone do not establish correctness; score with independent checks. If too few genuine faults exist, report that limitation.
- Do not silently inject faults into the AI-output study; a fault-injection study must be labelled separately.
- Execute code in a disposable offline environment without credentials or host-folder access, with resource limits.
- Before collecting participant data, agree consent, storage and withdrawal handling with the supervisor. Use participant codes, not names, in study files.
- Separate silent timed tasks from retrospective interviews. Timing during think-aloud sessions is descriptive, not an isolated interface-speed effect.
- For the master’s study, use the research task above to define the factors and comparisons in this pilot plan. Preregister one primary outcome and feasible scope after the literature review; do not add every possible model or interface variant.

REQUIRES
Inference compute, 8 developers and independent correctness review; no unseen-task or general model-performance claim.

--------------------

NON-TECHNICAL
- Prepare paper review packets containing a function, requirement and test report.
- Ask developers whether they would approve it, then interview them about trust in AI code.

DATA (planned)
12 printed Python packets + 8 developers
6 checked-correct and 6 faulty functions; 3 faults are missed by displayed tests. Hide labels and keep code identical between evidence-present/absent versions.


METHOD
Python-reading developers approve, reject or defer on disjoint packets. Compare decision errors and code reliance on tests versus independent inspection. No model execution or interface development.

OUTPUT
Review checklist and findings on misplaced trust.

BACHELOR
Task: Identify the checking steps students use when reviewing AI-generated code.
Data: 12 printed Python packets (planned)
6 checked-correct and 6 faulty functions; 3 faults are missed by displayed tests. Hide labels and keep code identical between evidence-present/absent versions.
Requires: Recruit developers. Reading code is needed; writing software is not.
Method: Use printed code/test packs in think-aloud sessions; classify missed checks and useful evidence. Plan 4 participant sessions.
Output: A review checklist grounded in observed mistakes.
Use relevant literature to justify the established approach; a new research contribution is not the aim of this proposal.

MASTER
Explain overreliance on plausible AI code despite available contradictory evidence. Contrast verification-cost and competence-belief explanations using task observations and follow-up interviews.
Intended contribution: A theory-based account of evidence use in AI-code review.

STARTING PAPERS
Buçinca et al. (2021) — To Trust or to Think: https://arxiv.org/abs/2102.09692
Derive a competing explanation based on verification effort and overreliance.
Vasconcelos et al. (2023) — Explanations Can Reduce Overreliance: https://arxiv.org/abs/2212.06823
Compare verification cost with trust as explanations of observed decisions.

Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points.

START HERE
Tools: LibreOffice Writer/Calc, audio recorder with consent; no programming required.
1. Prepare a pilot with 2 examples from: 12 prepared Python review packets + 8 developers. Write the task questions and a reference answer sheet.
2. Write a recruitment message, information sheet and consent form for the participants named above. Agree privacy handling with the supervisor before contact.
3. Pilot one session after approval; revise unclear questions, freeze the task sets and coding categories, then recruit the planned sample.
Literature search: AI generated code review trust test evidence qualitative scenario study
Study controls:
- Independent reviewer checks all correctness labels. Displayed passing tests may intentionally be incomplete; make no claim that all tests prove correctness.
- Before collecting participant data, agree consent, storage and withdrawal handling with the supervisor. Use participant codes, not names, in study files.
- Pilot separately, then freeze the questions and coding plan. Check objective answer keys independently; keep an audit trail of coding, including disagreements.
- For comparisons, counterbalance order and case assignment; do not show a person both versions of one case. Report participant-level findings, not repeated tasks as independent people.

REQUIRES
Recruit developers. Reading code is needed; writing software is not.

Bachelor: apply established methods and evaluate a practical solution or study. Master: position a research question in current scientific literature, investigate a mechanism or unresolved problem, and explain the contribution. Final scope is agreed with me.
Study sizes are proposed; participant recruitment and planned materials are not already arranged.