Overview

Given a reference image of an authentic Qatari cultural scene, an image caption, and an AI-generated image produced from that caption, participants develop systems that predict CRAI scores across five cultural accuracy dimensions. Systems are ranked by Spearman correlation against human-annotated gold scores.

The core challenge: our pilot study showed that GPT-4 as a judge achieves a Spearman correlation of 0.6250 against human annotators, with near-zero correlation on hallucination detection. Can you build a system that better aligns with human cultural judgment?

Task Description

Given three elements per instance — a reference image, a caption, and an AI-generated image produced from that caption — participants predict CRAI scores across five dimensions.

Input

One row per instance in captions.tsv:

id              image_id    version    caption
img_001_v1      img_001     v1         Generate an image of a Qatari man wearing a white thobe...
img_001_v2      img_001     v2         A man wearing traditional Gulf attire...

Each instance also has two associated images: a reference image (ref/img_001.jpg) and a generated image (generated/img_001_v1.png).

Output

A TSV file with one row per instance:

id            CRAI_CEA  CRAI_CC  CRAI_CS  CRAI_CI  CRAI_HP  CRAI_composite
img_001_v1    0.90      0.85     0.75     0.88     0.05     0.82
img_001_v2    0.60      0.55     0.50     0.70     0.05     0.58

Evaluation

CRAI Framework

The Cultural Representation Accuracy Index (CRAI) scores images across five dimensions:

CRAI_composite = 0.30 × CEA + 0.20 × CC + 0.20 × CS + 0.20 × CI − 0.10 × HP

Cultural Specificity (CS) uses fixed tiers:

Score interpretation: ≥ 0.85 highly accurate · 0.70–0.84 mostly accurate · 0.50–0.69 moderate · 0.30–0.49 weak · < 0.30 poor or misleading.

Dataset

The dataset consists of (reference image, caption, AI-generated image) triples grounded in Qatari culture. Each of 40 reference images has 5 caption versions varying in cultural specificity from fully Qatari-specific (v1) to entirely generic (v5).

Cultural categories covered: people and traditional attire, architecture and built environment, objects.

The full dataset, with gold labels for every split, is on Hugging Face, mirrored on Google Drive.

Baselines

The starter kit includes two ready-to-run baseline scripts.

# Run the GPT-4 baseline
python baselines/baseline_gpt4.py \
    --captions data/dev/captions.tsv \
    --ref_dir  data/dev/imgs/ref \
    --gen_dir  data/dev/imgs/generated \
    --output   predictions.tsv \
    --api_key  YOUR_OPENROUTER_KEY

# Evaluate locally
python evaluation/evaluate.py --gold data/dev/gold_human.tsv --pred predictions.tsv

Participation

You may use any model or pipeline. Approaches based on vision-language models, LLM-as-a-judge, multimodal embeddings, fine-tuned scorers, or novel combinations are all welcome.

Every participating team is expected to submit a short system description paper. See the submission guidelines.

Submission

The test phase has closed. Official results are on the Leaderboard page.

1. Produce predictions in the TSV format shown above.
2. Validate with the format checker:

   python format_checker/check_format.py --pred predictions.tsv --split dev
   

3. Evaluate locally with the scorer:

   python evaluation/evaluate.py --gold data/dev/gold_human.tsv --pred predictions.tsv
   

4. Zip as predictions.zip and upload to the Codabench leaderboard:

   zip predictions.zip predictions.tsv
   

All deadlines are Anywhere on Earth (AOE).

Submission limits: Development phase, 10/day. Test phase: a limited number of submissions per team; the best submission counts.

Questions

See the FAQ or email imageeval2026@gmail.com. The dataset is released for research use under CC BY-NC-SA 4.0.