Choosing which tests run first
Order automated tests so that failures are found earlier within a fixed CI time budget.
Technical
Bachelor’s · applied project
Final result
A test-ordering script and reproducible benchmark report.
Dataset + material
Flask 3.1.0 tests + 30 non-equivalent mutations
Record outcomes and durations, including survivors. finalise rankings using 10 history mutants before 20 evaluation mutants at non-overlapping mutation sites; history is artificial.
What to do
- Implement an established shortest-test-first order and display the resulting pytest run.
How it is tested
Use the pinned Flask tests and mutation fixture; measure time to first failure and check that reordering preserves results. Plan 4 participant sessions.
Before you start
Example tools · Python, Flask 3.1.0, pytest, pandas These are suggestions; equivalent tools are fine.
Access or people · Stable isolated test infrastructure, mutation checking and developers.
Research method · Build and test · 6 steps
- Read the starting sources and choose one established implementation method.
- Write the requirements, data fields, system diagram and test cases.
- Prepare Flask 3.1.0 tests + 30 non-equivalent mutations. Make the answer key and pass criteria before testing.
- Build a working version of a test-ordering script and reproducible benchmark report.
- Run function, integration and failure-case tests. Record each result.
- Run the practical evaluation and list the changes the system still needs.
Download the complete bachelor plan
Download bachelor instructionsMaster’s · research project
Final result
Five-order pytest scheduler, fixed replay dataset, NAPFD/APFDc benchmark and decision dashboard.
Dataset + material
Flask 3.1.0 tests + 30 non-equivalent mutations
Record outcomes and durations, including survivors. finalise rankings using 10 history mutants before 20 evaluation mutants at non-overlapping mutation sites; history is artificial.
What to do
- Implement original, seeded-random, shortest-first, historical-failure and change-coverage/runtime greedy pytest orders under one 25% wall-time budget.
- Build history and clean-suite line-coverage matrices without future outcomes; keep history and evaluation mutation sites non-overlapping and account for scheduler overhead.
- Verify full-run result preservation and no leakage; report NAPFD/APFDc, detected fraction, censored time-to-first-failure and policy-scored merge/defer decisions.
How it is tested
Verify full-run result preservation and no future-outcome leakage. Under the same 25% wall-time budget report NAPFD/APFDc, detected fraction, completed tests and right-censored time-to-first-failure. Four release-experienced developers compare dashboard/raw-log evidence and make policy-scored merge/defer decisions.
Before you start
Example tools · Python, Flask 3.1.0, pytest, pandas These are suggestions; equivalent tools are fine.
Access or people · Stable isolated test infrastructure, mutation checking and 4 developers.
Start with these three actions
- Check out Flask 3.1.0; run the unchanged suite and save the collected test order.
- Create one mutation, measure test durations and verify which tests detect it.
- Separate history from evaluation mutations and implement replay without reading future outcomes.
Files, questions and sources
Pilot question: Can earlier failure feedback improve developers’ understanding of what remains unverified?
Literature search: continuous integration test prioritisation developer feedback decisions
Empty CSV templates: data-plan.csv · evaluation.csv · literature-matrix.csv
Download starter instructionsResearch method · Design, build and test (DSR) · 7 steps
- Read the newest papers and list the closest existing systems.
- Write down the versions, fields, data split, case assignment, random seeds and correct answers for Flask 3.1.0 tests + 30 non-equivalent mutations.
- Draw the user workflow, data model and system architecture. List the requirements and pass criteria.
- Build a working version of five-order pytest scheduler, fixed replay dataset, napfd/apfdc benchmark and decision dashboard.
- Test every function, connection and failure case. Save the failed tests as well as the passed tests.
- Compare the system with the named alternative. Then run the user task or decision task in the assignment.
- Report the measured result, the failed cases and the design lessons another team can reuse.
Research question and sources
Research question: Investigate the trade-off between fast failures and incomplete assurance. Compare prioritization baselines on held-out mutation sites under equal wall-clock budgets; analyse censored failures and developers’ interpretation of partial coverage.
Research result: A reproducible account of prioritization benefits and decision risks under a fixed budget.
Current project literature
- Faster Feedback with AI? A Test Prioritization Study (2024) peer-reviewed
Compares modern and simple test-prioritization methods using change-based mutations. - Feature-oriented test case selection and prioritization during the evolution of highly-configurable systems (2024) peer-reviewed
Provides current evidence on selection overhead and failure detection under test budgets.
IS theory starting points
- Pan et al. (2021) — Test Selection and Prioritization: A Systematic Review — Compare prioritization features, baselines and cost-sensitive evaluation measures.
- Vessey & Galletta (1991) — Cognitive Fit — Test whether a representation helps one kind of task more than another.
Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points.
Search terms: continuous integration test prioritisation developer feedback decisions
Non-technical
Bachelor’s · applied project
Final result
A clearer test-report template and release checklist.
Material to create
12 CI reports
4 complete passes, 4 partial runs, 2 failures and 2 timeouts. Supply one explicit release policy and deadline; hide the reference decisions.
What to do
- Find which information developers need in a partial test-run report.
How it is tested
Use the paper run summaries and an explicit release policy; observe release decisions and document misunderstandings. Plan 4 participant sessions.
Before you start
Example tools · LibreOffice Writer/Calc, audio recorder with consent; no programming required. These are suggestions; equivalent tools are fine.
Access or people · Recruit developers; use prepared reports, no CI implementation.
Research method · Study and improve · 6 steps
- Read the starting sources and write the exact information problem.
- Prepare 12 CI reports and a separate answer sheet or coding sheet.
- Run one pilot and fix unclear questions.
- Collect the named evidence with consent.
- Group the findings with the stated categories and check the answer sheet.
- Produce a clearer test-report template and release checklist. List the three most useful changes.
Download the complete bachelor plan
Download bachelor instructionsMaster’s · research project
Final result
CI reporting checklist and analysis of merge decisions.
Material to create
12 CI reports + 8 developer interviews
4 complete passes, 4 partial runs, 2 failures and 2 timeouts. Supply one explicit release policy and deadline; hide the reference decisions.
What to do
- Prepare CI reports showing tests run, skipped, failed and timed out.
- Ask developers whether they would merge each change under a deadline.
How it is tested
Compare raw reports with identical evidence highlighting completion status. Count policy-unjustified merges and perceived risk, missing evidence and escalation reasoning; partial runs are not inherently unsafe.
Before you start
Example tools · LibreOffice Writer/Calc, audio recorder with consent; no programming required. These are suggestions; equivalent tools are fine.
Access or people · Recruit developers; use prepared reports, no CI implementation.
Start with these three actions
- Prepare a pilot with 2 examples from: 12 fictional CI reports + 8 developers. Write the task questions and a reference answer sheet.
- Write a recruitment message, information sheet and consent form for the participants named above. Agree privacy handling with the supervisor before contact.
- Pilot one session after approval; revise unclear questions, finalise the task sets and coding categories, then recruit the planned sample.
Files, questions and sources
Pilot question: Which missing CI evidence changes a developer’s decision to merge under a deadline?
Literature search: continuous integration test prioritisation developer feedback decisions qualitative scenario study
Interview prompts
- Would you merge this change now?
- Which behaviour remains unchecked?
- Which missing result would change your decision?
Empty CSV templates: study-plan.csv · coding-sheet.csv · literature-matrix.csv
Download starter instructionsResearch method · Study, compare and explain · 7 steps
- Read the newest papers and write one exact research question.
- State which people, cases or documents you will study and what you will compare.
- Write the selection rules, questions and analysis steps for 12 CI reports + 8 developer interviews.
- Run one pilot. Fix unclear questions or categories, then keep the guide unchanged.
- Collect the named interviews, cases or documents with consent.
- Analyse them with the stated comparison or coding method. Keep disagreements and missing data.
- Report the answer, the evidence and the practical output named in the assignment.
Research question and sources
Research question: Explain when partial test results are treated as sufficient release evidence. Use the same release policy across scenarios; compare verification effort, report representation and responsibility as other explanations.
Research result: A specific account of release confidence under incomplete testing evidence.
Current project literature
- Faster Feedback with AI? A Test Prioritization Study (2024) peer-reviewed
Compares modern and simple test-prioritization methods using change-based mutations. - Feature-oriented test case selection and prioritization during the evolution of highly-configurable systems (2024) peer-reviewed
Provides current evidence on selection overhead and failure detection under test budgets.
IS theory starting points
- Vasconcelos et al. (2023) — Explanations Can Reduce Overreliance — Compare verification cost with trust as explanations of observed decisions.
- Orlikowski & Gash (1994) — Technological Frames — Compare how roles interpret the purpose, operation and use of the same system.
Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points.
Search terms: continuous integration test prioritisation developer feedback decisions qualitative scenario study
Your degree, track and shortlist are saved in this browser.