When moderation labels disagree
Build or study a moderation process that shows disagreement instead of hiding it behind one label.
Technical
Bachelor’s · applied project
Final result
Working dashboard, reproducible baseline, test report and prioritised fixes.
Dataset
Fixed 3,000-record COUNTER subset
Select 1,000 records per language with a fixed seed, preserving label proportions and source groups. Keep a separate 20% test split.
What to do
- Build a multilingual moderation dashboard with a TF–IDF classifier, confidence warning and manual-review queue.
How it is tested
Use a fixed stratified subset of 3,000 COUNTER records. Compare the classifier with a majority-class baseline; report macro-F1, per-language F1, confusion matrices and calibration. Test the dashboard with 4 trained reviewers on 24 fixed cases.
Before you start
Example tools · Python, pandas, scikit-learn, Hugging Face Transformers, Streamlit, MLflow or DVC These are suggestions; equivalent tools are fine.
Access or people · Confirm dataset terms and ethics handling; recruit 4 trained reviewers.
Research method · Build and test · 6 steps
- Read the starting sources and choose one established implementation method.
- Write the requirements, data fields, system diagram and test cases.
- Prepare Fixed 3,000-record COUNTER subset. Make the answer key and pass criteria before testing.
- Build a working version of working dashboard, reproducible baseline, test report and prioritised fixes.
- Run function, integration and failure-case tests. Record each result.
- Run the practical evaluation and list the changes the system still needs.
Download the complete bachelor plan
Download bachelor instructionsMaster’s · research project
Final result
Reproducible training pipeline, calibrated review interface, model card and comparison report.
Dataset
COUNTER multilingual radical-content dataset
Use the public pseudonymised COUNTER release. Keep language, radicalisation level, call-for-action labels, named-entity fields and individual annotator labels. Pin the GitLab commit and dataset card. Use only this release.
What to do
- Preserve annotator-level labels and implement majority, multilingual transformer and label-distribution models on a source-safe fixed split.
- Calibrate probabilities and implement an explicit abstention policy; verify leakage, split integrity, deterministic inference and per-language behavior.
- Report macro-F1, per-language F1, Brier score, ECE, distribution distance and reviewer outcomes; separate model accuracy from escalation quality.
How it is tested
Use the published split, or a group split by conversation/source if the release requires a new split. Report macro-F1, per-language F1, Brier score, expected calibration error, Jensen–Shannon distance to annotator distributions and abstention coverage/error. In the reviewer study, compare majority-label output with disagreement-aware output on the same balanced cases; measure correct escalation, false escalation, verification requests, confidence and time.
Before you start
Example tools · Python, pandas, scikit-learn, Hugging Face Transformers, Streamlit, MLflow or DVC These are suggestions; equivalent tools are fine.
Access or people · Confirm dataset terms and ethics handling; recruit 8–12 trained reviewers or relevant professionals for the master evaluation.
Start with these three actions
- Clone and pin the COUNTER release; reproduce label counts, annotator agreement and the published split.
- Implement the majority TF–IDF baseline and calibration report before training a transformer.
- finalise the comparison, primary metric and abstention rule; then build the two reviewer views.
Files, questions and sources
Pilot question: When does showing label disagreement and calibrated uncertainty improve a moderator’s escalation decision?
Literature search: multilingual radical content detection annotator disagreement uncertainty calibration human review
Empty CSV templates: data-plan.csv · evaluation.csv · literature-matrix.csv
Download starter instructionsResearch method · Design, build and test (DSR) · 7 steps
- Read the newest papers and list the closest existing systems.
- Write down the versions, fields, data split, case assignment, random seeds and correct answers for COUNTER multilingual radical-content dataset.
- Draw the user workflow, data model and system architecture. List the requirements and pass criteria.
- Build a working version of reproducible training pipeline, calibrated review interface, model card and comparison report.
- Test every function, connection and failure case. Save the failed tests as well as the passed tests.
- Compare the system with the named alternative. Then run the user task or decision task in the assignment.
- Report the measured result, the failed cases and the design lessons another team can reuse.
Research question and sources
Research question: Test whether modelling the full annotator-label distribution and allowing abstention improves decisions compared with majority-label classification.
Research result: Evidence about when disagreement-aware moderation helps, including language-specific failure modes and a reusable design for human review under uncertainty.
Current project literature
- Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection (2025) peer-reviewed conference paper
Provides the public multilingual dataset, annotator-level disagreement and bias analysis used by the project. - Investigating radicalisation indicators in online extremist communities (2024) peer-reviewed workshop paper
Shows why labels must be treated as contextual and uncertain instead of direct diagnoses of a person.
IS theory starting points
- Riabi et al. (2025) — Beyond Dataset Creation — Use its dataset, annotation analysis and bias findings as the empirical starting point.
- de Kock & Hovy (2024) — Investigating radicalisation indicators — Bound the labels and avoid treating radicalisation as a simple, directly observable property.
Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points.
Search terms: multilingual radical content detection annotator disagreement uncertainty calibration human review
Non-technical
Bachelor’s · applied project
Final result
Revised coding guide, disagreement table and moderation-workflow recommendations.
Dataset + material
60-record COUNTER sample
Use 20 records per language, balanced across published levels and disagreement. Work only in languages the coders can assess reliably.
What to do
- Find where two coders interpret the same moderation rule differently and turn the findings into clearer guidance.
How it is tested
Dual-code 60 records with a fixed codebook. Report agreement by label and language, discuss every disagreement and revise the guide once.
Before you start
Example tools · LibreOffice Calc, Taguette or NVivo; no programming required. These are suggestions; equivalent tools are fine.
Access or people · Two trained coders; confirm dataset and ethics handling.
Research method · Study and improve · 6 steps
- Read the starting sources and write the exact information problem.
- Prepare 60-record COUNTER sample and a separate answer sheet or coding sheet.
- Run one pilot and fix unclear questions.
- Collect the named evidence with consent.
- Group the findings with the stated categories and check the answer sheet.
- Produce revised coding guide, disagreement table and moderation-workflow recommendations. List the three most useful changes.
Download the complete bachelor plan
Download bachelor instructionsMaster’s · research project
Final result
Coding manual, disagreement map and a concrete escalation-and-appeal policy.
Dataset + material
COUNTER annotation guide + 120-record stratified sample + workshop decisions
Select 40 English, 40 French and 40 Arabic records, balanced across published radicalisation levels and disagreement. Use translated excerpts only when validated; record coder rationale, confidence, escalation and disagreement resolution.
What to do
- Map how COUNTER’s annotation guide turns an ambiguous judgement into a platform label.
- Recode 120 stratified records with two trained coders and retain every disagreement.
- Run a structured workshop that compares three escalation rules: majority, any severe label and uncertainty-based review.
How it is tested
Conduct qualitative content analysis of the guide and coder rationales. Calculate agreement by language and label, then compare which records each escalation rule sends to review. Use 6–8 trained participants in one structured workshop; analyse the reasons for accepting or rejecting each rule.
Before you start
Example tools · LibreOffice Calc, Taguette or NVivo; no programming required. These are suggestions; equivalent tools are fine.
Access or people · Confirm dataset terms and ethics handling; recruit bilingual coders or limit the study to languages the team can assess reliably.
Start with these three actions
- Read the dataset paper and annotation guide; define the unit of analysis and sampling table.
- Pilot-code 12 records with two coders; revise and finalise the codebook before selecting the final 120.
- Predefine the three escalation rules and workshop questions; record decisions and disagreements separately.
Files, questions and sources
Pilot question: How do annotation rules and disagreement change what a moderation process escalates?
Literature search: content moderation annotation disagreement platform governance human review multilingual
Interview prompts
- What evidence makes this record clear or unclear?
- When should disagreement trigger human review?
- What explanation and appeal should a user receive?
Empty CSV templates: data-plan.csv · evaluation.csv · literature-matrix.csv
Download starter instructionsResearch method · Study, compare and explain · 7 steps
- Read the newest papers and write one exact research question.
- State which people, cases or documents you will study and what you will compare.
- Write the selection rules, questions and analysis steps for COUNTER annotation guide + 120-record stratified sample + workshop decisions.
- Run one pilot. Fix unclear questions or categories, then keep the guide unchanged.
- Collect the named interviews, cases or documents with consent.
- Analyse them with the stated comparison or coding method. Keep disagreements and missing data.
- Report the answer, the evidence and the practical output named in the assignment.
Research question and sources
Research question: Explain how annotation rules and escalation thresholds distribute uncertainty between automated systems, reviewers and users.
Research result: A process model and testable propositions about how label disagreement becomes a governance decision in AI-supported moderation.
Current project literature
- Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection (2025) peer-reviewed conference paper
Provides the public multilingual dataset, annotator-level disagreement and bias analysis used by the project. - Investigating radicalisation indicators in online extremist communities (2024) peer-reviewed workshop paper
Shows why labels must be treated as contextual and uncertain instead of direct diagnoses of a person.
IS theory starting points
- Riabi et al. (2025) — Beyond Dataset Creation — Ground the analysis in observed annotation variation and bias.
- Kokshagina et al. (2023) — To regulate or not to regulate — Connect moderation choices to platform regulation and public accountability.
Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points.
Search terms: content moderation annotation disagreement platform governance human review multilingual
Your degree, track and shortlist are saved in this browser.