Can AI recognise persuasion in Ukrainian wartime memes?
Not reliably. On 200 native Ukrainian memes, the best micro hierarchical F1 was 0.384 and macro F1 stayed below 0.20. Models often over-predicted persuasion, while explanations could agree on the same words and still assign the wrong rhetorical label.
The short version
- What happened
- Researchers collected 200 native Ukrainian memes and created a 158-item Ukrainian translation of an English benchmark, then tested four models without task-specific training.
- Who it affects
- Ukrainian journalists and researchers gain a culturally grounded dataset but still need native-speaking human judgement.
- What remains unproven
- The study is an unreviewed preprint and had no independent replication at publication.
Keep in mind
The study is an unreviewed preprint and had no independent replication at publication.
The native dataset has 200 memes and the translated control 158, limiting statistical power and cultural coverage.
Human agreement was moderate because persuasion labels are subjective and overlapping.
PTM-UA retains English-source imagery and may contain translation, OCR and rendering artefacts.
Only four zero-shot models were tested; the results do not establish performance after task-specific training or on future models.
Source analysis · 2 linked sources. Primary analysed source date: 9 October 2026.
Read the full report · 6 min The evidence, methods, implications and limitations
The answer: performance was too weak for automated judgement
The best reported micro hierarchical F1 on the 200 native Ukrainian memes was 0.384, achieved by GPT-4.1-mini with image and text. Macro F1, which gives more weight to rare categories, peaked at only 0.198. Qwen3-VL-8B's image-and-text result was 0.342 micro F1 and 0.193 macro F1. Those scores show partial recognition of rhetorical families, not reliable identification of the specific techniques present in a meme.
The models frequently over-predicted persuasion when visual and textual signals were combined, especially in emotionally charged wartime material. That matters because a moderation or monitoring system can turn a difficult cultural judgement into a false label at scale. The paper explicitly treats the labels as an analytical lens rather than assuming that Ukrainian grassroots memes are state propaganda. The results do not support using a model to decide that an individual post is propaganda or that its author is acting deceptively.[1][2]
Two datasets separated translation from native culture
UkrMeme contains 200 organically produced Ukrainian-language wartime memes collected from public Telegram channels and the SUCHO Meme Wall in early 2026. Two trained native speakers independently assigned labels from a 22-technique taxonomy plus a No Propaganda label, and a curator resolved disagreements. The second dataset, PTM-UA, began with an English propaganda-meme benchmark. Researchers extracted and removed English text, translated it into Ukrainian with GPT-5, rendered it back onto the image and manually filtered poor translations or damaged layouts, leaving 158 items.
That design creates a useful contrast. PTM-UA keeps the imagery and political discourse of an English benchmark while changing the written language. UkrMeme reflects Ukrainian wartime culture, humour and resilience. Performance was consistently higher on the translated set: GPT-4.1-mini reached 0.613 micro hierarchical F1 with image and text, compared with 0.384 on native memes. Translation therefore did not reproduce the cultural and rhetorical difficulty of authentic local material.[1][2]
What was measured and why agreement matters
The four zero-shot models were Qwen3-VL-4B-Instruct, Qwen3-VL-8B-Instruct, GPT-4.1-mini and Gemma 3 4B. Tests compared text-only, image-only and combined inputs. Hierarchical F1 awards partial credit when a predicted label and the gold label share a parent in the taxonomy, such as related emotional or credibility appeals. Even under that more forgiving metric, rare techniques remained difficult: macro F1 stayed below 0.30 on PTM-UA and below 0.20 on UkrMeme.
Human annotation was not perfectly settled. MASI-weighted Krippendorff's alpha was 0.406 and mean per-class Cohen's kappa was 0.404, both indicating moderate agreement. Agreement was strongest for No Propaganda, smears and appeals to fear or prejudice, and very low for loaded language. This is not merely annotator error; rhetorical categories overlap and context changes interpretation. A model evaluated against an uncertain boundary cannot be treated as discovering an objective fact that people themselves consistently recognise.[1]
Explanations did not guarantee a correct label
For the two open Qwen models, the researchers compared generated token explanations with Layer-wise Relevance Propagation, which traces internal relevance through the network. Verbal explanations tended to emphasise emotional or rhetorical cues, while internal relevance concentrated more on content words. Native multimodal memes showed higher median overlap between the two methods than translated examples, suggesting more stable cue selection in some settings.
But agreement between explanations did not mean the answer was right. In several low-scoring cases, both methods focused on the same words yet the model mapped those cues to the wrong technique—for example confusing a smear, loaded language, name-calling or an appeal to emotion. Other cases showed divergence that may reflect optical-character-recognition noise or translation artefacts. A plausible highlight or rationale is therefore not proof that the classification is faithful or correct.[1]
Impact on journalists, moderators and communities
For journalists and researchers, the dataset fills a real gap: most persuasion benchmarks are English-centred, while memes combine language, imagery, shared history and humour. Tools may help organise a corpus or surface items for review, but the low scores and uncertain labels argue for human analysis. Investigators should retain the original post, source and context rather than relying on extracted text or a model-generated category.
For platforms and public agencies, automated enforcement would be especially risky. A system that over-predicts persuasion could suppress satire, coping, political speech or documentation of war. The harms are not evenly distributed: lower-resource languages often receive weaker models and fewer appeal mechanisms. Any operational use should be limited to research or triage, with native-speaking reviewers, transparent uncertainty and a route for affected people to challenge a label.[1]
Evidence limits and what would change the assessment
Both datasets are small. Two hundred native memes cannot represent the range of Ukrainian regional language, political positions, platforms or changing wartime references. PTM-UA keeps English-source imagery and may preserve pretraining familiarity, while machine translation and re-rendering introduce artefacts. The model comparison is zero-shot and limited to four systems; task-specific training, new model versions or different prompts could change the ranking. The accessible paper does not report a funding or competing-interest statement.
Confidence would rise with a larger time-stratified dataset, more native annotators from varied backgrounds, clear treatment of disagreement and evaluation on newly collected material hidden from developers. Studies should separate descriptive research from moderation decisions and measure real false-positive consequences. For now, the strongest conclusion is constructive but cautious: culturally grounded Ukrainian data exposes weaknesses that translated English benchmarks hide, and neither a confident label nor an attractive explanation is enough to automate judgement.[1][2]