When moderation labels disagree — Non-technical
https://thesis.uya.no/proposals/when-moderation-labels-disagree/#nontechnical-track

BACHELOR
Final result: Revised coding guide, disagreement table and moderation-workflow recommendations.
Task: Find where two coders interpret the same moderation rule differently and turn the findings into clearer guidance.
Data: 60-record COUNTER sample (mixed)
Use 20 records per language, balanced across published levels and disagreement. Work only in languages the coders can assess reliably.
Requires: Two trained coders; confirm dataset and ethics handling.
Method: Dual-code 60 records with a fixed codebook. Report agreement by label and language, discuss every disagreement and revise the guide once.

STEPS
Study and improve
1. Read the starting sources and write the exact information problem.
2. Prepare 60-record COUNTER sample and a separate answer sheet or coding sheet.
3. Run one pilot and fix unclear questions.
4. Collect the named evidence with consent.
5. Group the findings with the stated categories and check the answer sheet.
6. Produce revised coding guide, disagreement table and moderation-workflow recommendations. List the three most useful changes.

MASTER
Final result: Coding manual, disagreement map and a concrete escalation-and-appeal policy.
Research question: Explain how annotation rules and escalation thresholds distribute uncertainty between automated systems, reviewers and users.
Research result: A process model and testable propositions about how label disagreement becomes a governance decision in AI-supported moderation.

STEPS
Study, compare and explain
1. Read the newest papers and write one exact research question.
2. State which people, cases or documents you will study and what you will compare.
3. Write the selection rules, questions and analysis steps for COUNTER annotation guide + 120-record stratified sample + workshop decisions.
4. Run one pilot. Fix unclear questions or categories, then keep the guide unchanged.
5. Collect the named interviews, cases or documents with consent.
6. Analyse them with the stated comparison or coding method. Keep disagreements and missing data.
7. Report the answer, the evidence and the practical output named in the assignment.

CURRENT PROJECT LITERATURE
Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection (2025, peer-reviewed conference paper): https://aclanthology.org/2025.coling-main.578/
Provides the public multilingual dataset, annotator-level disagreement and bias analysis used by the project.

Investigating radicalisation indicators in online extremist communities (2024, peer-reviewed workshop paper): https://aclanthology.org/2024.woah-1.1/
Shows why labels must be treated as contextual and uncertain instead of direct diagnoses of a person.

IS THEORY STARTING POINTS
Riabi et al. (2025) — Beyond Dataset Creation: https://aclanthology.org/2025.coling-main.578/
Ground the analysis in observed annotation variation and bias.
Kokshagina et al. (2023) — To regulate or not to regulate: https://aisel.aisnet.org/jit/vol38/iss2/5/
Connect moderation choices to platform regulation and public accountability.

Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points.

PILOT QUESTION
How do annotation rules and disagreement change what a moderation process escalates?

EXAMPLE TOOLS
LibreOffice Calc, Taguette or NVivo; no programming required.
Equivalent tools are fine.

FIRST THREE ACTIONS
1. Read the dataset paper and annotation guide; define the unit of analysis and sampling table.
2. Pilot-code 12 records with two coders; revise and finalise the codebook before selecting the final 120.
3. Predefine the three escalation rules and workshop questions; record decisions and disagreements separately.

INTERVIEW QUESTIONS
- What evidence makes this record clear or unclear?
- When should disagreement trigger human review?
- What explanation and appeal should a user receive?

LITERATURE SEARCH
content moderation annotation disagreement platform governance human review multilingual

DATA
COUNTER annotation guide + 120-record stratified sample + workshop decisions (mixed)
Select 40 English, 40 French and 40 Arabic records, balanced across published radicalisation levels and disagreement. Use translated excerpts only when validated; record coder rationale, confidence, escalation and disagreement resolution.
COUNTER public dataset: https://gitlab.inria.fr/ariabi/counter-dataset-public

HOW TO TEST IT
Conduct qualitative content analysis of the guide and coder rationales. Calculate agreement by language and label, then compare which records each escalation rule sends to review. Use 6–8 trained participants in one structured workshop; analyse the reasons for accepting or rejecting each rule.

FINAL RESULT
Coding manual, disagreement map and a concrete escalation-and-appeal policy.

ACCESS OR PEOPLE
Confirm dataset terms and ethics handling; recruit bilingual coders or limit the study to languages the team can assess reliably.

TEMPLATES
https://thesis.uya.no/starters/when-moderation-labels-disagree/nontechnical/data-plan.csv
https://thesis.uya.no/starters/when-moderation-labels-disagree/nontechnical/evaluation.csv
https://thesis.uya.no/starters/when-moderation-labels-disagree/nontechnical/literature-matrix.csv
Use participant codes instead of names or email addresses.