Retora vs Human Message Evaluations

Total-index audience analysis with dimension diagnostics

Executive Summary

Retora coverage
100%
Participant-message rows
640 / 640
Total-index MAE
1.12
Average signed bias
-0.66
Retora coverage
100%
Participant-message rows
640 / 640
Total-index MAE
1.12
Average signed bias
-0.66

Plain-language readout: On the 1-7 total message-evaluation index, Retora lands within 1 point of the human average in 58.0% of participant-message cases. Using a looser band, it is within 1.5 points in 74.4% of cases and within 2 points in 87.2%.

Overall, Retora is not wildly off, but it is systematically more negative: its total score is on average 0.66 points lower than humans. The average miss is 1.12 points on the same 1-7 scale.

Trust pattern: Retora is least accurate for Low trust participants (MAE 1.22, 320 participant-messages) and most accurate for Medium trust participants (MAE 0.96). The strongest directional trust-bin mismatch is for Low trust, where Retora scores are on average lower than humans by 0.85 points. This is a descriptive subgroup pattern; the report avoids treating a nearly flat trend line as evidence of a strong linear trust relationship.

Where Retora Matches And Misses

Where Retora matches and misses: The finder scans gender, age, gender-by-age, progressive position, trust, message topic, and topic-by-gender groups. Only groups with at least 30 participant-message rows are ranked.

Best matched groups:

  • taxation and redistribution, Male (Topic x gender; 86 participant-messages): MAE 0.95, Retora lower by 0.66 points on average
  • Medium trust (Trust; 228 participant-messages): MAE 0.96, Retora lower by 0.39 points on average
  • Male, 35+ (Gender x age; 244 participant-messages): MAE 1.03, Retora lower by 0.54 points on average
  • Progressive (Progressive position; 364 participant-messages): MAE 1.04, Retora lower by 0.54 points on average
  • taxation and redistribution (Message topic; 160 participant-messages): MAE 1.07, Retora lower by 0.66 points on average

Largest deviations:

  • Male, 18-34 (Gender x age; 100 participant-messages): MAE 1.23, Retora lower by 0.99 points on average
  • Mixed (Progressive position; 232 participant-messages): MAE 1.23, Retora lower by 0.81 points on average
  • Low trust (Trust; 320 participant-messages): MAE 1.22, Retora lower by 0.85 points on average
  • Conservative / low progressive (Progressive position; 44 participant-messages): MAE 1.21, Retora lower by 0.91 points on average
  • 18-34 (Age; 212 participant-messages): MAE 1.21, Retora lower by 0.80 points on average

Direction of mismatch: The strongest directional mismatch is Male, 18-34 (Gender x age), where Retora is lower than humans by 0.99 points on average.

These are descriptive performance patterns from the current dataset. They identify where Retora is closer or further from human ratings, but they should not be read as causal evidence about why the mismatch happens.

Group type Group Participant-messages Participants Human mean Retora mean Total-index MAE Signed bias Within 1 point
Gender x age Male, 18-34 100 25 5.08 4.09 1.23 -0.99 53.0%
Progressive position Mixed 232 57 4.72 3.91 1.23 -0.81 52.6%
Trust Low trust 320 79 4.80 3.94 1.22 -0.85 54.1%
Progressive position Conservative / low progressive 44 11 4.10 3.19 1.21 -0.91 54.5%
Age 18-34 212 52 5.00 4.20 1.21 -0.80 54.2%
Topic x gender immigration, Female 73 72 4.91 4.19 1.20 -0.72 56.2%
Topic x gender taxation and redistribution, Female 73 72 4.93 4.26 1.20 -0.67 47.9%
Gender x age Female, 18-34 112 27 4.92 4.30 1.18 -0.63 55.4%
Topic x gender gender pay gap, Male 86 84 5.07 4.41 1.17 -0.65 57.0%
Topic x gender climate change, Male 86 84 4.52 3.80 1.16 -0.72 60.5%
Gender Female 292 72 4.92 4.25 1.16 -0.67 53.8%
Trust High trust 92 23 5.02 4.35 1.16 -0.68 51.1%
Message topic gender pay gap 160 157 5.12 4.48 1.15 -0.64 55.0%
Gender x age Female, 35+ 180 45 4.92 4.23 1.15 -0.69 52.8%
Message topic climate change 160 157 4.57 3.89 1.13 -0.68 59.4%
Message topic immigration 160 157 4.79 4.11 1.13 -0.68 58.8%
Topic x gender gender pay gap, Female 73 72 5.21 4.57 1.13 -0.65 53.4%
Topic x gender climate change, Female 73 72 4.63 4.00 1.11 -0.63 57.5%
Gender Male 344 84 4.75 4.07 1.09 -0.67 61.3%
Age 35+ 428 105 4.73 4.13 1.08 -0.59 59.8%
Topic x gender immigration, Male 86 84 4.70 4.04 1.07 -0.66 60.5%
Message topic taxation and redistribution 160 157 4.80 4.14 1.07 -0.66 58.8%
Progressive position Progressive 364 89 4.97 4.43 1.04 -0.54 61.8%
Gender x age Male, 35+ 244 59 4.61 4.07 1.03 -0.54 64.8%
Trust Medium trust 228 56 4.77 4.38 0.96 -0.39 66.2%
Topic x gender taxation and redistribution, Male 86 84 4.70 4.04 0.95 -0.66 67.4%

Multivariate Drivers Of Deviation

The model below explains absolute Retora-human deviation on the total message-evaluation index. It adjusts audience traits and message design in the same model, with cluster-robust standard errors by participant.

Multivariate drivers: This section asks which factors are associated with larger absolute Retora-human deviation after adjusting for other audience traits and message design. The outcome is absolute error on the total message-evaluation index.

The adjusted OLS model uses 636 participant-message rows from 156 participants. Its adjusted R-squared is 0.073, so these factors explain some, but not most, of the case-level mismatch.

The largest incremental factor in this model is Age x gender (partial R-squared 0.036). Among audience traits, the largest is Age x gender (partial R-squared 0.036).

These estimates are descriptive associations, not causal effects. Standard errors are clustered by participant because each participant rated multiple messages.

Model rows Participants Adjusted R-squared Residual SE
636 156 0.073 0.82

Total Evaluation Index

The total index is the average of existing message-rating dimensions. Human and Retora totals are paired row by row, using only dimensions the human participant actually rated.

Audience Index Diagnostics

Progressive index mean Progressive index SD Trust index mean Trust index SD
5.97 1.86 3.30 2.37
Index Group Participant-messages Participants Total-index MAE Signed bias
Trust High trust 92 23 1.16 -0.68
Trust Low trust 320 79 1.22 -0.85
Trust Medium trust 228 56 0.96 -0.39
Progressive position Conservative / low progressive 44 11 1.21 -0.91
Progressive position Mixed 232 57 1.23 -0.81
Progressive position Progressive 364 89 1.04 -0.54

Human Vs Retora Dimension Averages

These interactive bars compare average human and Retora ratings by client-readable audience groups. Error bars show 95% normal confidence intervals around the mean.

Dimension Diagnostics

The charts below decompose the total index into the original moral-rating dimensions. They are diagnostic detail rather than the headline audience result.

Message Quality Matching

Message quality pattern: Retora is furthest from humans for weak messages (MAE 1.27, within 1 point in 53.5% of cases) and closest for decent messages (MAE 1.02, within 1 point in 61.6% of cases).

The strongest quality-level directional mismatch is for weak messages, where Retora scores are on average lower than humans by 0.95 points. This section asks whether presumed message quality changes the size and direction of Retora-human deviation.

Quality Participant-messages Participants Human mean Retora mean Total-index MAE Signed bias Within 1 point
weak 213 133 4.86 3.91 1.27 -0.95 53.5%
decent 216 132 4.84 4.25 1.02 -0.59 61.6%
strong 211 126 4.75 4.31 1.06 -0.45 58.8%

Message Diagnostics

Best And Worst Matched Participant-Messages

Section Participant Topic Format Quality Message ID Message Human total Retora total Abs. error Signed bias
Closest predictions 5483702ffdf99b24c654b810 climate_change pitch decent climate_change_pitch_authority_q2_v1 A clean-energy transition needs stable rules, not last-minute improvisation. Clear targets help firms invest, workers... 4.25 4.25 0.00 0.00
Closest predictions 58ff31a1d10e2b000108579e immigration motto strong immigration_motto_fairness_q3_v1 When newcomers follow the process, timely decisions and shared duties make fair treatment visible. 3.75 3.75 0.00 0.00
Closest predictions 5affe1f4331e8100018f1d45 climate_change motto weak climate_change_motto_freedom_q1_v2 Less dependence on fossil fuels means people have more energy choices. 4.00 4.00 0.00 0.00
Closest predictions 5cfde288bab656000198b943 gender_pay_gap motto strong gender_pay_gap_motto_fairness_q3_v1 Salary audits and clear promotion rules expose hidden bias before it shapes earnings. 4.50 4.50 0.00 0.00
Closest predictions 5d2466b24813f6000188f4b4 taxation_and_redistribution motto weak taxation_and_redistribution_motto_harm_q1_v1 Public services should reduce hardship for families under pressure. 3.25 3.25 0.00 0.00
Closest predictions 5d88dec3439d210016835821 gender_pay_gap motto decent gender_pay_gap_motto_decency_q2_v1 Responsible employers check pay decisions and correct unequal treatment. 5.00 5.00 0.00 0.00
Closest predictions 5d9750fe74d24900145633a7 climate_change pitch weak climate_change_pitch_fairness_q1_v1 Climate action should be fair. People who pollute more should probably do more. Ordinary households should not be lef... 3.75 3.75 0.00 0.00
Closest predictions 5e89ba09ec9b3364b1fc0bd1 immigration pitch strong immigration_pitch_fairness_q3_v2 A credible migration system should not punish patience or hide costs in a few districts. It should give newcomers lan... 4.75 4.75 0.00 0.00
Closest predictions 5ee92b925f0f482bb1e73668 taxation_and_redistribution pitch weak taxation_and_redistribution_pitch_freedom_q1_v2 Progressive taxation can help people have more options. If housing, care, and education are out of reach, life gets n... 4.50 4.50 0.00 0.00
Closest predictions 5f102f0212ea4f000a8ac9da climate_change motto decent climate_change_motto_fairness_q2_v2 A clean-energy transition must share costs according to responsibility and ability to pay. 3.75 3.75 0.00 0.00
Closest predictions 5f3ec93efbd299158253f283 gender_pay_gap pitch strong gender_pay_gap_pitch_freedom_q3_v2 Biased pay decisions do not just reduce income; they reduce room to decide. They can shape whether someone can move, ... 5.25 5.25 0.00 0.00
Closest predictions 5fa4f94e535f3b1aa628e0ab taxation_and_redistribution pitch strong taxation_and_redistribution_pitch_authority_q3_v2 A serious economy cannot run on crumbling services and optional tax discipline for the wealthiest. Clear progressive ... 4.00 4.00 0.00 0.00
Closest predictions 60e0bf29c82295dfeed33944 immigration motto strong immigration_motto_authority_q3_v2 Good rules make migration safer, more orderly, and easier for communities to manage. 4.75 4.75 0.00 0.00
Closest predictions 657619485d6869976045a220 gender_pay_gap motto strong gender_pay_gap_motto_decency_q3_v1 A workplace with integrity does not let bias hide inside the payslip. 4.75 4.75 0.00 0.00
Closest predictions 6658b1507df7606690e4d685 gender_pay_gap motto decent gender_pay_gap_motto_decency_q2_v2 Basic workplace standards require honest reviews of gender-based pay gaps. 4.50 4.50 0.00 0.00
Closest predictions 665d92a608fef982937744df immigration pitch decent immigration_pitch_belonging_q2_v2 Communities are healthier when newcomers are not left on the edge. Practical support helps them learn the language, f... 4.00 4.00 0.00 0.00
Closest predictions 66bb63dab2b110041a61af00 gender_pay_gap motto weak gender_pay_gap_motto_freedom_q1_v2 No one should have fewer options because pay decisions are biased. 4.50 4.50 0.00 0.00
Closest predictions 66ef27cedb6307178a650d79 immigration motto decent immigration_motto_authority_q2_v2 Orderly routes and consistent decisions make humane migration easier to sustain. 3.25 3.25 0.00 0.00
Closest predictions 6720c26681a4d34fea38c6ef gender_pay_gap motto decent gender_pay_gap_motto_decency_q2_v2 Basic workplace standards require honest reviews of gender-based pay gaps. 5.25 5.25 0.00 0.00
Closest predictions 6783ebc2e89b75d66d952f41 immigration motto weak immigration_motto_fairness_q1_v1 Migrants should have a fair chance to build a stable life. 3.50 3.50 0.00 0.00
Closest predictions 678ff81c8c6b56f93355e0bb immigration motto strong immigration_motto_authority_q3_v2 Good rules make migration safer, more orderly, and easier for communities to manage. 4.25 4.25 0.00 0.00
Closest predictions 681472b129f35dad5de67837 gender_pay_gap pitch strong gender_pay_gap_pitch_freedom_q3_v1 Lower pay narrows choices: leaving a bad role, saving, studying, or balancing care with ambition. Pay equality gives ... 5.00 5.00 0.00 0.00
Closest predictions 698b47e0a785e079268c0615 immigration motto strong immigration_motto_fairness_q3_v1 When newcomers follow the process, timely decisions and shared duties make fair treatment visible. 3.75 3.75 0.00 0.00
Closest predictions 6992095be7942a3fc333a078 taxation_and_redistribution motto strong taxation_and_redistribution_motto_harm_q3_v1 One illness, rent rise, or job loss should not push a family into crisis. 5.25 5.25 0.00 0.00
Closest predictions 6992095be7942a3fc333a078 immigration motto strong immigration_motto_authority_q3_v1 A humane system needs published routes, firm timelines, and support councils can prepare for. 4.75 4.75 0.00 0.00
Closest predictions 69cc28cd29af8d640be609c5 immigration motto strong immigration_motto_fairness_q3_v2 A fair system asks newcomers to contribute and gives them real tools to do so. 4.50 4.50 0.00 0.00
Closest predictions 69cc28cd29af8d640be609c5 climate_change motto decent climate_change_motto_harm_q2_v1 Cleaner power and better buildings reduce risks from heat, floods, and dirty air. 4.00 4.00 0.00 0.00
Closest predictions 69ec0801854129d6e09b1a0b taxation_and_redistribution motto strong taxation_and_redistribution_motto_fairness_q3_v2 A fair economy asks great wealth to support the foundations that made success possible. 3.75 3.75 0.00 0.00
Closest predictions 69ec0801854129d6e09b1a0b immigration pitch decent immigration_pitch_belonging_q2_v1 Humane migration policy should help newcomers become active members of local life. Language learning, work, and commu... 4.50 4.50 0.00 0.00
Closest predictions 69ed80361ac67e060ad6e85e immigration motto strong immigration_motto_fairness_q3_v1 When newcomers follow the process, timely decisions and shared duties make fair treatment visible. 4.75 4.75 0.00 0.00
Closest predictions 69eeab43b5b5e976fb68e27a climate_change motto decent climate_change_motto_fairness_q2_v2 A clean-energy transition must share costs according to responsibility and ability to pay. 4.00 4.00 0.00 0.00
Closest predictions 69efb336c58d4100c1b4a231 immigration pitch strong immigration_pitch_belonging_q3_v1 Integration works best when people stop living parallel lives. Language classes, work routes, and local meeting place... 3.25 3.25 0.00 0.00
Closest predictions 69f5b0fdb36c29c2c4ae7c07 gender_pay_gap motto strong gender_pay_gap_motto_harm_q3_v1 One lower offer can compound into smaller raises, weaker savings, and a thinner pension. 5.25 5.25 0.00 0.00
Closest predictions 69fa27f897bf4caf2d505098 taxation_and_redistribution motto strong taxation_and_redistribution_motto_freedom_q3_v2 Progressive taxes can buy ordinary people room to move, learn, and rebuild. 5.25 5.25 0.00 0.00
Closest predictions 69fdeef22b9541ff637e2c91 gender_pay_gap motto decent gender_pay_gap_motto_decency_q2_v2 Basic workplace standards require honest reviews of gender-based pay gaps. 5.50 5.50 0.00 0.00
Closest predictions 69fdeef22b9541ff637e2c91 climate_change motto decent climate_change_motto_fairness_q2_v2 A clean-energy transition must share costs according to responsibility and ability to pay. 3.75 3.75 0.00 0.00
Closest predictions 69fe89c667d2daf3d480d753 gender_pay_gap motto weak gender_pay_gap_motto_freedom_q1_v2 No one should have fewer options because pay decisions are biased. 4.75 4.75 0.00 0.00
Closest predictions 69ff71b9ee7aed92a102e70a immigration motto strong immigration_motto_fairness_q3_v1 When newcomers follow the process, timely decisions and shared duties make fair treatment visible. 4.00 4.00 0.00 0.00
Closest predictions 69ff71b9ee7aed92a102e70a climate_change motto weak climate_change_motto_fairness_q1_v1 Big polluters should pay more toward the clean-energy transition. 4.50 4.50 0.00 0.00
Closest predictions 6a00893df8dbc36e2be09c1c gender_pay_gap motto weak gender_pay_gap_motto_fairness_q1_v2 Pay should be fair when people are doing similar work. 3.75 3.75 0.00 0.00
Closest predictions 6a032e7f10303a5ab57cb700 climate_change pitch decent climate_change_pitch_freedom_q2_v1 Renewable energy can give households and communities more control over their future. Cleaner power reduces dependence... 3.75 3.75 0.00 0.00
Closest predictions 6a042d9260c03aa95f88992e taxation_and_redistribution pitch strong taxation_and_redistribution_pitch_freedom_q3_v1 A person is not truly choosing if every option is blocked by rent, care costs, or a school place they cannot access. ... 5.50 5.50 0.00 0.00
Closest predictions 6a0af8db0162308ca5e3f59b taxation_and_redistribution pitch decent taxation_and_redistribution_pitch_authority_q2_v2 A well-run tax system sets expectations, closes loopholes, and funds visible public priorities. That makes stronger i... 4.25 4.25 0.00 0.00
Closest predictions 6a0ec280a3320a348d9630aa climate_change pitch weak climate_change_pitch_fairness_q1_v2 Cleaner energy costs money, and those costs should be shared better. Big polluters should contribute more than famili... 3.25 3.25 0.00 0.00
Closest predictions 6a1629cc5dcd6ea2c000c9af immigration pitch decent immigration_pitch_authority_q2_v2 People trust migration policy more when the process is consistent. Clear routes, realistic timelines, and planned loc... 5.00 5.00 0.00 0.00
Closest predictions 6a16ea99c41f24c9371f4d00 immigration pitch weak immigration_pitch_belonging_q1_v2 Integration should bring people together. If newcomers and local residents rarely meet, trust becomes harder. A more ... 5.50 5.50 0.00 0.00
Closest predictions 6a2189a94939fa537f7e0f1a gender_pay_gap motto decent gender_pay_gap_motto_freedom_q2_v1 Equal pay helps workers save, move, retrain, or leave bad jobs. 4.75 4.75 0.00 0.00
Closest predictions 6a232d1401eefd2809ecd27d climate_change motto decent climate_change_motto_fairness_q2_v2 A clean-energy transition must share costs according to responsibility and ability to pay. 3.00 3.00 0.00 0.00
Closest predictions 6a25865d2d6177946644c54c immigration pitch strong immigration_pitch_fairness_q3_v1 Fair treatment is more than a welcoming phrase. It means clear duties for newcomers, timely decisions from the state,... 4.50 4.50 0.00 0.00
Largest deviations 5f6c69c8ef3322000b04fc6a immigration motto weak immigration_motto_fairness_q1_v2 A fair migration system should treat people with basic respect. 7.00 1.00 6.00 -6.00
Largest deviations 6a2199e15b03d651347b9916 climate_change motto weak climate_change_motto_harm_q1_v2 Clean energy can help reduce climate risks for families and communities. 6.00 1.50 4.50 -4.50
Largest deviations 5755c957eb80c4000741a9ce gender_pay_gap motto strong gender_pay_gap_motto_harm_q3_v1 One lower offer can compound into smaller raises, weaker savings, and a thinner pension. 1.00 5.25 4.25 4.25
Largest deviations 6a1fd86b35130b24ecb78cbb immigration motto strong immigration_motto_harm_q3_v2 Humane entry routes and planned services keep vulnerable people out of avoidable danger. 6.00 2.00 4.00 -4.00
Largest deviations 55fc1a7c77f33b0010cb229f immigration pitch decent immigration_pitch_harm_q2_v1 A humane migration policy can reduce avoidable suffering without ignoring local capacity. Clear routes, timely decisi... 5.25 1.50 3.75 -3.75
Largest deviations 57a0d81d81f331000139d989 immigration pitch strong immigration_pitch_harm_q3_v2 People seeking safety should not be pushed into uncertainty while councils improvise support at the last minute. Huma... 5.25 1.75 3.50 -3.50
Largest deviations 69fb7b9b615d3fefc0ec2abb immigration motto weak immigration_motto_fairness_q1_v2 A fair migration system should treat people with basic respect. 7.00 3.50 3.50 -3.50
Largest deviations 6a183c4b7720c4cf5c72ee47 climate_change pitch strong climate_change_pitch_freedom_q3_v2 A country that depends on expensive imported fuels gives up too much control over its future. Renewables, storage, an... 7.00 3.50 3.50 -3.50