Ranking under each sentiment pathway
Same cached inputs for every pathway (MR_OFFLINE=1): RT, BOM, CinemaScore and the same 1,381 comments; only the per-comment sentiment/stance step differs. Rank = position in the shortlist (bucket first, then score). Rank change = live rank minus pathway rank (positive = moves up). Labels for accuracy are mine (Grok Bot), single annotator. Live index.html still uses siebert+opinion.
Cell = #rank · score · change vs live (↑ moves up) · bucket · R = Reddit community score. Colour = bucket (Watch Depends Skip). Click a cell for that movie's comments with each pathway's verdict and rationale. The live shortlist on the home page is unchanged (siebert+opinion).
Stance pathways ask: did the commenter like or recommend the film? (positive / negative / mixed / no verdict, JSON, temperature 0). Mixed and no-verdict count as neutral. OpenRouter spend for these evaluations and runs: $13.63. Not run: Local Qwen2.5-3B stance
Stance prompt (shared by every LLM pathway, stance-v1)
You classify a Reddit comment from a movie's discussion thread by the commenter's VERDICT ON THE FILM ITSELF: did they like it / would they recommend it?
Judge the verdict, not the tone or vocabulary.
- Praise of dark, scary, violent, gross, sad or disturbing content is POSITIVE ("the villain is terrifying", "that scene was brutal, incredible", "I cried the whole time, beautiful").
- Hating a character, calling a character evil/a psycho, or describing a horrifying scene is NOT a negative verdict unless they criticise the film.
- Polite or soft disappointment is NEGATIVE ("I really wanted to like it", "it fell flat for me", "beautiful but I didn't care").
- Clear praise and clear criticism with no overall lean is MIXED. If they lean one way overall ("I liked it despite flaws"), pick that side.
- Plot discussion, theories, quotes, jokes, trivia, comments about the book/director/marketing without a judgement of this film, or not having seen it: NO_VERDICT.
Reply with JSON only: {"verdict": "positive" | "negative" | "mixed" | "no_verdict", "rationale": "<max 20 words>"}NLI hypotheses
- positive: The writer liked this movie or recommends it, even if they describe dark, scary or violent scenes.
- negative: The writer thinks this movie is bad or was disappointed by it.
- mixed: The writer has mixed feelings about this movie, praising some parts and criticising others.
- no_verdict: The writer is only discussing the plot, a character, a scene or trivia, without judging the movie.
Challenge set by type (correct / n)
| pathway | dark words, positive verdict | polite words, negative verdict | dark description, no verdict | mixed |
|---|---|---|---|---|
| VADER (old v1) | 28/39 | 10/19 | 1/11 | 0/2 |
| siebert+opinion (LIVE) | 35/39 | 17/19 | 2/11 | 0/2 |
| Local NLI stance (deberta-v3-large) | 26/39 | 15/19 | 5/11 | 1/2 |
| GPT-6.1 Sol | 36/39 | 18/19 | 11/11 | 1/2 |
| Claude Opus 5.5 | 36/39 | 16/19 | 10/11 | 2/2 |
| Claude Sonnet 5.5 | 35/39 | 15/19 | 11/11 | 2/2 |
| Gemini 4 Argon | 36/39 | 15/19 | 10/11 | 2/2 |
| Grok 4.7 | 36/39 | 16/19 | 10/11 | 1/2 |
| GPT-6 Luna (cheap) | 36/39 | 18/19 | 10/11 | 2/2 |
| Claude Haiku 5.5 (cheap) | 36/39 | 16/19 | 10/11 | 2/2 |
| Gemini 3.5 Flash-Lite (cheap) | 35/39 | 14/19 | 10/11 | 2/2 |
Data: compare_pathways.json · stance_eval.json (every labelled comment, every pathway) · challenge labels