When moderation labels disagree
P1 · Spring 2027
https://thesis.uya.no/proposals/when-moderation-labels-disagree/

Build or study a moderation process that shows disagreement instead of hiding it behind one label.

TECHNICAL IMPLEMENTATION
- Preserve annotator-level labels and implement majority, multilingual transformer and label-distribution models on a source-safe fixed split.
- Calibrate probabilities and implement an explicit abstention policy; verify leakage, split integrity, deterministic inference and per-language behavior.
- Report macro-F1, per-language F1, Brier score, ECE, distribution distance and reviewer outcomes; separate model accuracy from escalation quality.

TECHNICAL

FINAL RESULT
Reproducible training pipeline, calibrated review interface, model card and comparison report.

WHAT TO DO
- Preserve annotator-level labels and implement majority, multilingual transformer and label-distribution models on a source-safe fixed split.
- Calibrate probabilities and implement an explicit abstention policy; verify leakage, split integrity, deterministic inference and per-language behavior.
- Report macro-F1, per-language F1, Brier score, ECE, distribution distance and reviewer outcomes; separate model accuracy from escalation quality.

DATA (public)
COUNTER multilingual radical-content dataset
Use the public pseudonymised COUNTER release. Keep language, radicalisation level, call-for-action labels, named-entity fields and individual annotator labels. Pin the GitLab commit and dataset card. Use only this release.
COUNTER public dataset: https://gitlab.inria.fr/ariabi/counter-dataset-public

HOW TO TEST IT
Use the published split, or a group split by conversation/source if the release requires a new split. Report macro-F1, per-language F1, Brier score, expected calibration error, Jensen–Shannon distance to annotator distributions and abstention coverage/error. In the reviewer study, compare majority-label output with disagreement-aware output on the same balanced cases; measure correct escalation, false escalation, verification requests, confidence and time.

BACHELOR
Final result: Working dashboard, reproducible baseline, test report and prioritised fixes.
Task: Build a multilingual moderation dashboard with a TF–IDF classifier, confidence warning and manual-review queue.
Data: Fixed 3,000-record COUNTER subset (public)
Select 1,000 records per language with a fixed seed, preserving label proportions and source groups. Keep a separate 20% test split.
Requires: Confirm dataset terms and ethics handling; recruit 4 trained reviewers.
Method: Use a fixed stratified subset of 3,000 COUNTER records. Compare the classifier with a majority-class baseline; report macro-F1, per-language F1, confusion matrices and calibration. Test the dashboard with 4 trained reviewers on 24 fixed cases.

STEPS
Build and test
1. Read the starting sources and choose one established implementation method.
2. Write the requirements, data fields, system diagram and test cases.
3. Prepare Fixed 3,000-record COUNTER subset. Make the answer key and pass criteria before testing.
4. Build a working version of working dashboard, reproducible baseline, test report and prioritised fixes.
5. Run function, integration and failure-case tests. Record each result.
6. Run the practical evaluation and list the changes the system still needs.

MASTER
Final result: Reproducible training pipeline, calibrated review interface, model card and comparison report.
Research question: Test whether modelling the full annotator-label distribution and allowing abstention improves decisions compared with majority-label classification.
Research result: Evidence about when disagreement-aware moderation helps, including language-specific failure modes and a reusable design for human review under uncertainty.

STEPS
Design, build and test (DSR)
1. Read the newest papers and list the closest existing systems.
2. Write down the versions, fields, data split, case assignment, random seeds and correct answers for COUNTER multilingual radical-content dataset.
3. Draw the user workflow, data model and system architecture. List the requirements and pass criteria.
4. Build a working version of reproducible training pipeline, calibrated review interface, model card and comparison report.
5. Test every function, connection and failure case. Save the failed tests as well as the passed tests.
6. Compare the system with the named alternative. Then run the user task or decision task in the assignment.
7. Report the measured result, the failed cases and the design lessons another team can reuse.

CURRENT PROJECT LITERATURE
Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection (2025, peer-reviewed conference paper): https://aclanthology.org/2025.coling-main.578/
Provides the public multilingual dataset, annotator-level disagreement and bias analysis used by the project.

Investigating radicalisation indicators in online extremist communities (2024, peer-reviewed workshop paper): https://aclanthology.org/2024.woah-1.1/
Shows why labels must be treated as contextual and uncertain instead of direct diagnoses of a person.

IS THEORY STARTING POINTS
Riabi et al. (2025) — Beyond Dataset Creation: https://aclanthology.org/2025.coling-main.578/
Use its dataset, annotation analysis and bias findings as the empirical starting point.
de Kock & Hovy (2024) — Investigating radicalisation indicators: https://aclanthology.org/2024.woah-1.1/
Bound the labels and avoid treating radicalisation as a simple, directly observable property.

Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points.

START WITH THESE THREE ACTIONS
1. Clone and pin the COUNTER release; reproduce label counts, annotator agreement and the published split.
2. Implement the majority TF–IDF baseline and calibration report before training a transformer.
3. finalise the comparison, primary metric and abstention rule; then build the two reviewer views.

EXAMPLE TOOLS
Python, pandas, scikit-learn, Hugging Face Transformers, Streamlit, MLflow or DVC
Equivalent tools are fine.

LITERATURE SEARCH
multilingual radical content detection annotator disagreement uncertainty calibration human review

ACCESS OR PEOPLE
Confirm dataset terms and ethics handling; recruit 8–12 trained reviewers or relevant professionals for the master evaluation.

--------------------

NON-TECHNICAL

FINAL RESULT
Coding manual, disagreement map and a concrete escalation-and-appeal policy.

WHAT TO DO
- Map how COUNTER’s annotation guide turns an ambiguous judgement into a platform label.
- Recode 120 stratified records with two trained coders and retain every disagreement.
- Run a structured workshop that compares three escalation rules: majority, any severe label and uncertainty-based review.

DATA (mixed)
COUNTER annotation guide + 120-record stratified sample + workshop decisions
Select 40 English, 40 French and 40 Arabic records, balanced across published radicalisation levels and disagreement. Use translated excerpts only when validated; record coder rationale, confidence, escalation and disagreement resolution.
COUNTER public dataset: https://gitlab.inria.fr/ariabi/counter-dataset-public

HOW TO TEST IT
Conduct qualitative content analysis of the guide and coder rationales. Calculate agreement by language and label, then compare which records each escalation rule sends to review. Use 6–8 trained participants in one structured workshop; analyse the reasons for accepting or rejecting each rule.

BACHELOR
Final result: Revised coding guide, disagreement table and moderation-workflow recommendations.
Task: Find where two coders interpret the same moderation rule differently and turn the findings into clearer guidance.
Data: 60-record COUNTER sample (mixed)
Use 20 records per language, balanced across published levels and disagreement. Work only in languages the coders can assess reliably.
Requires: Two trained coders; confirm dataset and ethics handling.
Method: Dual-code 60 records with a fixed codebook. Report agreement by label and language, discuss every disagreement and revise the guide once.

STEPS
Study and improve
1. Read the starting sources and write the exact information problem.
2. Prepare 60-record COUNTER sample and a separate answer sheet or coding sheet.
3. Run one pilot and fix unclear questions.
4. Collect the named evidence with consent.
5. Group the findings with the stated categories and check the answer sheet.
6. Produce revised coding guide, disagreement table and moderation-workflow recommendations. List the three most useful changes.

MASTER
Final result: Coding manual, disagreement map and a concrete escalation-and-appeal policy.
Research question: Explain how annotation rules and escalation thresholds distribute uncertainty between automated systems, reviewers and users.
Research result: A process model and testable propositions about how label disagreement becomes a governance decision in AI-supported moderation.

STEPS
Study, compare and explain
1. Read the newest papers and write one exact research question.
2. State which people, cases or documents you will study and what you will compare.
3. Write the selection rules, questions and analysis steps for COUNTER annotation guide + 120-record stratified sample + workshop decisions.
4. Run one pilot. Fix unclear questions or categories, then keep the guide unchanged.
5. Collect the named interviews, cases or documents with consent.
6. Analyse them with the stated comparison or coding method. Keep disagreements and missing data.
7. Report the answer, the evidence and the practical output named in the assignment.

CURRENT PROJECT LITERATURE
Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection (2025, peer-reviewed conference paper): https://aclanthology.org/2025.coling-main.578/
Provides the public multilingual dataset, annotator-level disagreement and bias analysis used by the project.

Investigating radicalisation indicators in online extremist communities (2024, peer-reviewed workshop paper): https://aclanthology.org/2024.woah-1.1/
Shows why labels must be treated as contextual and uncertain instead of direct diagnoses of a person.

IS THEORY STARTING POINTS
Riabi et al. (2025) — Beyond Dataset Creation: https://aclanthology.org/2025.coling-main.578/
Ground the analysis in observed annotation variation and bias.
Kokshagina et al. (2023) — To regulate or not to regulate: https://aisel.aisnet.org/jit/vol38/iss2/5/
Connect moderation choices to platform regulation and public accountability.

Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points.

START WITH THESE THREE ACTIONS
1. Read the dataset paper and annotation guide; define the unit of analysis and sampling table.
2. Pilot-code 12 records with two coders; revise and finalise the codebook before selecting the final 120.
3. Predefine the three escalation rules and workshop questions; record decisions and disagreements separately.

EXAMPLE TOOLS
LibreOffice Calc, Taguette or NVivo; no programming required.
Equivalent tools are fine.

LITERATURE SEARCH
content moderation annotation disagreement platform governance human review multilingual

ACCESS OR PEOPLE
Confirm dataset terms and ethics handling; recruit bilingual coders or limit the study to languages the team can assess reliably.