Choosing which tests run first Personal proposal by Per-Arne Andersen S6 · Spring 2027 https://thesis.uya.no/proposals/test-what-matters-first/ Order automated tests so that failures are found earlier within a fixed CI time budget. TECHNICAL - Measure individual Flask tests and create one controlled code mutation per run. - Implement shortest-test-first and previous-failure-first ordering. DATA (mixed) Flask 3.1.0 tests + 30 non-equivalent mutations Record outcomes and durations, including survivors. Freeze rankings using 10 history mutants before 20 evaluation mutants at disjoint mutation sites; history is artificial. Flask 3.1.0: https://github.com/pallets/flask/releases/tag/3.1.0 METHOD Compare shortest-first, historical-failure-first and recorded pytest order at 25% of clean-suite wall time. Count completed tests, faults detected, time-to-first-failure and timeouts. Four developers compare dashboard/raw-log merge or defer decisions. OUTPUT Test scheduler, replay dataset and benchmark. BACHELOR Task: Implement an established shortest-test-first order and display the resulting pytest run. Data: Flask 3.1.0 tests + 30 non-equivalent mutations (mixed) Record outcomes and durations, including survivors. Freeze rankings using 10 history mutants before 20 evaluation mutants at disjoint mutation sites; history is artificial. Requires: Stable isolated test infrastructure, mutation checking and developers. Method: Use the pinned Flask tests and mutation fixture; measure time to first failure and check that reordering preserves results. Plan 4 participant sessions. Output: A test-ordering script and reproducible benchmark report. Use relevant literature to justify the established approach; a new research contribution is not the aim of this proposal. MASTER Investigate the trade-off between fast failures and incomplete assurance. Compare prioritization baselines on held-out mutation sites under equal wall-clock budgets; analyse censored failures and developers’ interpretation of partial coverage. Intended contribution: A reproducible account of prioritization benefits and decision risks under a fixed budget. STARTING PAPERS Pan et al. (2021) — Test Selection and Prioritization: A Systematic Review: https://arxiv.org/abs/2106.13891 Compare prioritization features, baselines and cost-sensitive evaluation measures. Vessey & Galletta (1991) — Cognitive Fit: https://pubsonline.informs.org/doi/10.1287/isre.2.1.63 Test whether a representation helps one kind of task more than another. Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points. START HERE Tools: Python, Flask 3.1.0, pytest, pandas 1. Check out Flask 3.1.0; run the unchanged suite and save the collected test order. 2. Create one mutation, measure test durations and verify which tests detect it. 3. Separate history from evaluation mutations and implement replay without reading future outcomes. Literature search: continuous integration test prioritisation developer feedback decisions Study controls: - Include startup and fixture time in the budget; right-censor undetected failures at budget rather than treating them as fast successes. - Verify outcomes are stable under reordered execution. No future outcomes may inform rankings. - Dashboard and raw logs contain identical evidence; the dashboard exposes untested work clearly. - Before collecting participant data, agree consent, storage and withdrawal handling with the supervisor. Use participant codes, not names, in study files. - Separate silent timed tasks from retrospective interviews. Timing during think-aloud sessions is descriptive, not an isolated interface-speed effect. - For the master’s study, use the research task above to define the factors and comparisons in this pilot plan. Preregister one primary outcome and feasible scope after the literature review; do not add every possible model or interface variant. REQUIRES Stable isolated test infrastructure, mutation checking and 4 developers. -------------------- NON-TECHNICAL - Prepare CI reports showing tests run, skipped, failed and timed out. - Ask developers whether they would merge each change under a deadline. DATA (planned) 12 CI reports + 8 developer interviews 4 complete passes, 4 partial runs, 2 failures and 2 timeouts. Supply one explicit release policy and deadline; hide the reference decisions. METHOD Compare raw reports with identical evidence highlighting completion status. Count policy-unjustified merges and code risk, missing evidence and escalation reasoning; partial runs are not inherently unsafe. OUTPUT CI reporting checklist and analysis of merge decisions. BACHELOR Task: Find which information developers need in a partial test-run report. Data: 12 CI reports (planned) 4 complete passes, 4 partial runs, 2 failures and 2 timeouts. Supply one explicit release policy and deadline; hide the reference decisions. Requires: Recruit developers; use prepared reports, no CI implementation. Method: Use the paper run summaries and an explicit release policy; observe release decisions and document misunderstandings. Plan 4 participant sessions. Output: A clearer test-report template and release checklist. Use relevant literature to justify the established approach; a new research contribution is not the aim of this proposal. MASTER Explain when partial test results are treated as sufficient release evidence. Use the same release policy across scenarios; compare verification effort, report representation and responsibility as rival explanations. Intended contribution: An explanatory model of release confidence under incomplete testing. STARTING PAPERS Vasconcelos et al. (2023) — Explanations Can Reduce Overreliance: https://arxiv.org/abs/2212.06823 Compare verification cost with trust as explanations of observed decisions. Orlikowski & Gash (1994) — Technological Frames: https://dl.acm.org/doi/10.1145/196734.196745 Compare how roles interpret the purpose, operation and use of the same system. Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points. START HERE Tools: LibreOffice Writer/Calc, audio recorder with consent; no programming required. 1. Prepare a pilot with 2 examples from: 12 fictional CI reports + 8 developers. Write the task questions and a reference answer sheet. 2. Write a recruitment message, information sheet and consent form for the participants named above. Agree privacy handling with the supervisor before contact. 3. Pilot one session after approval; revise unclear questions, freeze the task sets and coding categories, then recruit the planned sample. Literature search: continuous integration test prioritisation developer feedback decisions qualitative scenario study Study controls: - Define acceptable exceptions in the policy before scoring decisions; do not invent an approval rule after seeing participants’ answers. - Before collecting participant data, agree consent, storage and withdrawal handling with the supervisor. Use participant codes, not names, in study files. - Pilot separately, then freeze the questions and coding plan. Check objective answer keys independently; keep an audit trail of coding, including disagreements. - For comparisons, counterbalance order and case assignment; do not show a person both versions of one case. Report participant-level findings, not repeated tasks as independent people. REQUIRES Recruit developers; use prepared reports, no CI implementation. Bachelor: apply established methods and evaluate a practical solution or study. Master: position a research question in current scientific literature, investigate a mechanism or unresolved problem, and explain the contribution. Final scope is agreed with me. Study sizes are proposed; participant recruitment and planned materials are not already arranged.