When moderation labels disagree — Technical
https://thesis.uya.no/proposals/when-moderation-labels-disagree/#technical-track

BACHELOR
Final result: Working dashboard, reproducible baseline, test report and prioritised fixes.
Task: Build a multilingual moderation dashboard with a TF–IDF classifier, confidence warning and manual-review queue.
Data: Fixed 3,000-record COUNTER subset (public)
Select 1,000 records per language with a fixed seed, preserving label proportions and source groups. Keep a separate 20% test split.
Requires: Confirm dataset terms and ethics handling; recruit 4 trained reviewers.
Method: Use a fixed stratified subset of 3,000 COUNTER records. Compare the classifier with a majority-class baseline; report macro-F1, per-language F1, confusion matrices and calibration. Test the dashboard with 4 trained reviewers on 24 fixed cases.

STEPS
Build and test
1. Read the starting sources and choose one established implementation method.
2. Write the requirements, data fields, system diagram and test cases.
3. Prepare Fixed 3,000-record COUNTER subset. Make the answer key and pass criteria before testing.
4. Build a working version of working dashboard, reproducible baseline, test report and prioritised fixes.
5. Run function, integration and failure-case tests. Record each result.
6. Run the practical evaluation and list the changes the system still needs.

MASTER
Final result: Reproducible training pipeline, calibrated review interface, model card and comparison report.
Research question: Test whether modelling the full annotator-label distribution and allowing abstention improves decisions compared with majority-label classification.
Research result: Evidence about when disagreement-aware moderation helps, including language-specific failure modes and a reusable design for human review under uncertainty.

STEPS
Design, build and test (DSR)
1. Read the newest papers and list the closest existing systems.
2. Write down the versions, fields, data split, case assignment, random seeds and correct answers for COUNTER multilingual radical-content dataset.
3. Draw the user workflow, data model and system architecture. List the requirements and pass criteria.
4. Build a working version of reproducible training pipeline, calibrated review interface, model card and comparison report.
5. Test every function, connection and failure case. Save the failed tests as well as the passed tests.
6. Compare the system with the named alternative. Then run the user task or decision task in the assignment.
7. Report the measured result, the failed cases and the design lessons another team can reuse.

CURRENT PROJECT LITERATURE
Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection (2025, peer-reviewed conference paper): https://aclanthology.org/2025.coling-main.578/
Provides the public multilingual dataset, annotator-level disagreement and bias analysis used by the project.

Investigating radicalisation indicators in online extremist communities (2024, peer-reviewed workshop paper): https://aclanthology.org/2024.woah-1.1/
Shows why labels must be treated as contextual and uncertain instead of direct diagnoses of a person.

IS THEORY STARTING POINTS
Riabi et al. (2025) — Beyond Dataset Creation: https://aclanthology.org/2025.coling-main.578/
Use its dataset, annotation analysis and bias findings as the empirical starting point.
de Kock & Hovy (2024) — Investigating radicalisation indicators: https://aclanthology.org/2024.woah-1.1/
Bound the labels and avoid treating radicalisation as a simple, directly observable property.

Search Scopus or Web of Science and ACM Digital Library using the topic query, then follow citations to the thesis start date. Record searches and compare methods, data, findings and limitations in literature-matrix.csv. Use that review to confirm or revise the gap and choose a current comparator. The linked papers are starting points.

PILOT QUESTION
When does showing label disagreement and calibrated uncertainty improve a moderator’s escalation decision?

EXAMPLE TOOLS
Python, pandas, scikit-learn, Hugging Face Transformers, Streamlit, MLflow or DVC
Equivalent tools are fine.

FIRST THREE ACTIONS
1. Clone and pin the COUNTER release; reproduce label counts, annotator agreement and the published split.
2. Implement the majority TF–IDF baseline and calibration report before training a transformer.
3. finalise the comparison, primary metric and abstention rule; then build the two reviewer views.

LITERATURE SEARCH
multilingual radical content detection annotator disagreement uncertainty calibration human review

DATA
COUNTER multilingual radical-content dataset (public)
Use the public pseudonymised COUNTER release. Keep language, radicalisation level, call-for-action labels, named-entity fields and individual annotator labels. Pin the GitLab commit and dataset card. Use only this release.
COUNTER public dataset: https://gitlab.inria.fr/ariabi/counter-dataset-public

HOW TO TEST IT
Use the published split, or a group split by conversation/source if the release requires a new split. Report macro-F1, per-language F1, Brier score, expected calibration error, Jensen–Shannon distance to annotator distributions and abstention coverage/error. In the reviewer study, compare majority-label output with disagreement-aware output on the same balanced cases; measure correct escalation, false escalation, verification requests, confidence and time.

FINAL RESULT
Reproducible training pipeline, calibrated review interface, model card and comparison report.

ACCESS OR PEOPLE
Confirm dataset terms and ethics handling; recruit 8–12 trained reviewers or relevant professionals for the master evaluation.

TEMPLATES
https://thesis.uya.no/starters/when-moderation-labels-disagree/technical/data-plan.csv
https://thesis.uya.no/starters/when-moderation-labels-disagree/technical/evaluation.csv
https://thesis.uya.no/starters/when-moderation-labels-disagree/technical/literature-matrix.csv
Use participant codes instead of names or email addresses.