Reviewing AI-generated code
Measure whether test evidence helps developers reject faulty AI-generated Python functions.
Technical
Bachelor’s · applied project
Final result
A review prototype and concrete usability fixes.
Dataset + material
HumanEval/0–39 + Qwen Coder completions
finalise model revision and prompt; save one greedy completion per task. Prepare independent requirement-based audit tests before generation; retain failed outputs.
HumanEval ↗ · Qwen2.5-Coder-7B-Instruct ↗ · Transformers generation settings ↗
What to do
- Build a code-review screen with links to tests and source lines.
How it is tested
Use the fixed HumanEval completions and independent audit tests; observe reviewers detecting and explaining faults. Plan 4 participant sessions.
Before you start
Example tools · Qwen2.5-Coder-7B-Instruct, Python, HumanEval, Streamlit These are suggestions; equivalent tools are fine.
Access or people · Inference compute, developers and independent correctness review; no unseen-task or general model-performance claim.
Research method · Build and test · 6 steps
- Read the starting sources and choose one established implementation method.
- Write the requirements, data fields, system diagram and test cases.
- Prepare HumanEval/0–39 + Qwen Coder completions. Make the answer key and pass criteria before testing.
- Build a working version of a review prototype and concrete usability fixes.
- Run function, integration and failure-case tests. Record each result.
- Run the practical evaluation and list the changes the system still needs.
Download the complete bachelor plan
Download bachelor instructionsMaster’s · research project
Final result
Review screen, completions and acceptance-error results.
Dataset + material
HumanEval/0–39 + Qwen Coder completions
finalise model revision and prompt; save one greedy completion per task. Prepare independent requirement-based audit tests before generation; retain failed outputs.
HumanEval ↗ · Qwen2.5-Coder-7B-Instruct ↗ · Transformers generation settings ↗
What to do
- finalise HumanEval completions with model/tokenizer SHA, runtime, quantization, prompt, greedy decoding, seed and output limit; run them only in an isolated sandbox.
- Compare raw pytest logs with line-linked requirement/test evidence while keeping code and tests identical; create independent audit tests and a fault taxonomy.
- Report public-test pass rate separately from false acceptance/rejection, fault detection, evidence navigation and time; verify capture determinism and test independence.
How it is tested
Report the 40-completion published-test pass fraction separately. Eight developers review 10 non-overlapping cases using either raw pytest logs or line-linked requirement and test evidence, with code and executed tests unchanged. Score false acceptance/rejection against independently verified faults, plus time and requests for more checking.
Before you start
Example tools · Qwen2.5-Coder-7B-Instruct, Python, HumanEval, Streamlit These are suggestions; equivalent tools are fine.
Access or people · Inference compute, 8 developers and independent correctness review; no unseen-task or general model-performance claim.
Start with these three actions
- Download HumanEval and the named model; record revisions, prompt and decoding settings. Confirm access to inference compute and record model, software and hardware versions.
- Generate tasks 0–2 and run the published tests in an isolated environment with no credentials.
- Prepare the same code-review layout in both conditions; show either raw pytest logs or line-linked requirement and test evidence while keeping code and executed tests unchanged.
Files, questions and sources
Pilot question: How does test evidence change developers’ acceptance of AI-generated functions?
Literature search: AI generated code review trust test evidence
Empty CSV templates: data-plan.csv · evaluation.csv · literature-matrix.csv
Download starter instructionsResearch method · Design, build and test (DSR) · 7 steps
- Read the newest papers and list the closest existing systems.
- Write down the versions, fields, data split, case assignment, random seeds and correct answers for HumanEval/0–39 + Qwen Coder completions.
- Draw the user workflow, data model and system architecture. List the requirements and pass criteria.
- Build a working version of review screen, completions and acceptance-error results.
- Test every function, connection and failure case. Save the failed tests as well as the passed tests.
- Compare the system with the named alternative. Then run the user task or decision task in the assignment.
- Report the measured result, the failed cases and the design lessons another team can reuse.
Research question and sources
Research question: Investigate whether evidence links improve detection of AI-code faults or reinforce misleading tests. Use independent audit tests and unchanged generated outputs; analyse natural error types and reviewer verification behaviour separately.
Research result: An evaluation linking test adequacy, traceability and human review effectiveness.
Current project literature
- LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation (2024) peer-reviewed
Shows that test-mediated interaction can improve evaluation of AI-generated code. - Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation (2024) peer-reviewed
Demonstrates that executable generated code can still contain substantial API misuse.
IS theory starting points
- Liu et al. (2023) — Is Your Code Generated by ChatGPT Really Correct? — Distinguish passing a public test suite from independently assessed correctness.
- Vasconcelos et al. (2023) — Explanations Can Reduce Overreliance — Compare verification cost with trust as explanations of observed decisions.
Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points.
Search terms: AI generated code review trust test evidence
Non-technical
Bachelor’s · applied project
Final result
A review checklist grounded in observed mistakes.
Material to create
12 printed Python packets
6 checked-correct and 6 faulty functions; 3 faults are missed by displayed tests. Hide labels and keep code identical between evidence-present/absent versions.
What to do
- Identify the checking steps Python-capable final-year IS students use when reviewing AI-generated code.
How it is tested
Use printed code/test packs in think-aloud sessions; classify missed checks and useful evidence. Plan 4 participant sessions.
Before you start
Example tools · LibreOffice Writer/Calc, audio recorder with consent; no programming required. These are suggestions; equivalent tools are fine.
Access or people · Recruit Python-capable final-year IS students. Reading code is required; writing software is not.
Research method · Study and improve · 6 steps
- Read the starting sources and write the exact information problem.
- Prepare 12 printed Python packets and a separate answer sheet or coding sheet.
- Run one pilot and fix unclear questions.
- Collect the named evidence with consent.
- Group the findings with the stated categories and check the answer sheet.
- Produce a review checklist grounded in observed mistakes. List the three most useful changes.
Download the complete bachelor plan
Download bachelor instructionsMaster’s · research project
Final result
Review checklist and findings on misplaced trust.
Material to create
12 printed Python packets + 8 Python-capable final-year IS students
6 checked-correct and 6 faulty functions; 3 faults are missed by displayed tests. Hide labels and keep code identical between evidence-present/absent versions.
What to do
- Prepare paper review packets containing a function, requirement and test report.
- Ask developers whether they would approve it, then interview them about trust in AI code.
How it is tested
Python-capable final-year IS students approve, reject or defer on non-overlapping packets. Compare per-person decision errors and reliance on displayed tests versus independent inspection; no model execution or interface development.
Before you start
Example tools · LibreOffice Writer/Calc, audio recorder with consent; no programming required. These are suggestions; equivalent tools are fine.
Access or people · Recruit 8 Python-capable final-year IS students. Reading code is required; writing software is not.
Start with these three actions
- Prepare a pilot with 2 of the 12 printed Python packets. Recruit 8 Python-capable final-year IS students for the full study; write the task questions and independent reference answer sheet.
- Write a recruitment message, information sheet and consent form for the participants named above. Agree privacy handling with the supervisor before contact.
- Pilot one session after approval; revise unclear questions, finalise the task sets and coding categories, then recruit the planned sample.
Files, questions and sources
Pilot question: When do reviewers rely on test results despite remaining gaps in verification?
Literature search: AI generated code review trust test evidence qualitative scenario study
Interview prompts
- Would you approve this function and why?
- Which behaviour do the tests actually cover?
- What would you check before relying on the AI output?
Empty CSV templates: study-plan.csv · coding-sheet.csv · literature-matrix.csv
Download starter instructionsResearch method · Study, compare and explain · 7 steps
- Read the newest papers and write one exact research question.
- State which people, cases or documents you will study and what you will compare.
- Write the selection rules, questions and analysis steps for 12 printed Python packets + 8 Python-capable final-year IS students.
- Run one pilot. Fix unclear questions or categories, then keep the guide unchanged.
- Collect the named interviews, cases or documents with consent.
- Analyse them with the stated comparison or coding method. Keep disagreements and missing data.
- Report the answer, the evidence and the practical output named in the assignment.
Research question and sources
Research question: Explain overreliance on plausible AI code among novice reviewers despite contradictory evidence. Contrast verification-cost and competence-belief explanations using task observations and follow-up interviews.
Research result: A specific account of evidence use in novice AI-code review.
Current project literature
- LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation (2024) peer-reviewed
Shows that test-mediated interaction can improve evaluation of AI-generated code. - Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation (2024) peer-reviewed
Demonstrates that executable generated code can still contain substantial API misuse.
IS theory starting points
- Buçinca et al. (2021) — To Trust or to Think — Derive a competing explanation based on verification effort and overreliance.
- Vasconcelos et al. (2023) — Explanations Can Reduce Overreliance — Compare verification cost with trust as explanations of observed decisions.
Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points.
Search terms: AI generated code review trust test evidence qualitative scenario study
Your degree, track and shortlist are saved in this browser.