📚 Task Documentation

Overview

This documentation provides all the details needed to understand, implement, and participate in the POLAR shared task. It includes dataset format, annotation scheme, task variants, evaluation metrics, and guidelines for submission.

🧪 Task Definition

Participants must develop models to detect and interpret attitude polarization in user comments from news discussions across multiple languages and topics. Given a news comment and context, the system should predict:

• Polarization Category (binary or multi-class depending on subtask)
• Token-Level Polarization Cues (for interpretability)

🧩 Subtasks

• 🔹 Subtask A: Polarization Classification
  Goal: Classify whether a comment expresses a polarized stance.
  Input: News comment + metadata (language, event, topic)
  Output: Label – Polarized or Not Polarized
• 🔹 Subtask B: Multi-Topic, Multi-Language Classification
  Goal: Classify the type of polarization (e.g., political, religious, ethnic)
  Output: Multi-class label (e.g., Political, Religious, Ethnic, None)
  Languages: English, German, Amharic, Hausa
• 🔹 Subtask C: Token-Level Cue Identification
  Goal: Highlight which tokens or phrases contribute to perceived polarization
  Output: Token-level binary labels (0: neutral, 1: polarizing cue)

🗃️ Dataset

• Languages: English, German, Amharic, Hausa
• Domains: News comments on real-world events (conflicts, elections, crises)
• Annotations: Human-labeled for both comment-level and token-level polarization
• Sources: Multilingual news portals, social commentary archives
• Structure:

🏷️ Annotation Guidelines

Polarized comments show strong alignment, antagonism, or emotional extremes toward social groups, ideologies, or policies. Cues include slurs, framing, emphasis, exaggeration, selective blame, etc. Native annotators provided culturally grounded judgments for each language.

A full annotation guideline document (PDF) will be made available on the Resources page.

🧮 Evaluation

• Subtask A and B:
  Accuracy, F1 Score (macro & weighted), Class-wise precision & recall
• Subtask C:
  Token-level F1 Score, Span-level Match Score (Jaccard), Optional human judgment for interpretability

💾 Baseline Models

• Multilingual BERT / XLM-R
• AfroXLM-R for low-resource languages
• Rule-based polarity patterns (for comparison)

Baseline code and training scripts will be published on our GitHub repository.

🚀 Submission Format

Prediction file: JSONL format
Required fields: id, label, and optionally cue_spans
Each team may submit up to 3 runs per subtask.

Submissions accepted via CodaLab during evaluation phase

📅 Timeline

Phase	Date
31 March 2025	Call for Participation Opens / Task Proposals Due
8 August 2025	Trial Data Released
1 September 2025	Training Data Released
1 December 2025	Test Data Released (internal deadline; not for public release)
10 January 2026	System Submission Deadline / Evaluation Start
31 January 2026	Evaluation Results Released / Evaluation End
February 2026	System Paper Submission Deadline
March 2026	Notification of Acceptance
April 2026	Camera-Ready Papers Due
Summer 2026	SemEval-2026 Workshop at [Conference Location TBD]

📎 Resources

• Annotation Guidelines (PDF)
• Baseline Code (GitHub)
• Frequently Asked Questions (FAQ)
• Related Research Papers

📬 Contact

For any questions, reach out to us at:
📧 polarization-semeval-2026-organisers@googlegroups.com
Or open an issue on GitHub Discussions