How sentence-model conclusions move under unreliable labels
Manual labels come from Pausanias book 3.rtf (Greta/Rosie). GPT labels are the active three-way original-myth-history-other run using gpt-5.4-mini.
| Manual Label | GPT Historical | GPT Mythic | GPT Other | Total |
|---|---|---|---|---|
| Historical | 196 | 14 | 23 | 233 |
| Mythic | 24 | 217 | 58 | 299 |
| Other / not highlighted | 92 | 74 | 255 | 421 |
| Mixed mythic+historical | 3 | 3 | 0 | 6 |
| Total | 315 | 308 | 336 | 959 |
The scenario models all use the same lemma TF-IDF logistic-regression setup as the current sentence analysis. Rows tagged other are not used in mythic-vs-historical fitting unless a scenario turns them into a mythic or historical sample.
| Scenario | Sample | Historical | Mythic | Features | Accuracy | Hist F1 | Myth F1 | Downweighted | Status |
|---|---|---|---|---|---|---|---|---|---|
| GPT labels on manual Book 3 scope Uses the active GPT-5.4-mini three-way tags, restricted to sentences that have manual Book 3 labels. |
623 | 315 | 308 | 925 | 0.615 | 0.605 | 0.625 | Available | |
| Manual labels only Uses Greta/Rosie manual labels; unhighlighted and mixed rows are dropped for mythic-vs-historical fitting. |
532 | 233 | 299 | 745 | 0.699 | 0.636 | 0.744 | Available | |
| Agreement-only labels Keeps only sentences where manual and GPT labels exactly agree as mythic or historical. |
413 | 196 | 217 | 634 | 0.644 | 0.611 | 0.673 | Available | |
| GPT labels, disagreements downweighted Uses GPT labels but gives manual/GPT disagreements 35% of the weight of agreement rows. |
623 | 315 | 308 | 925 | 0.596 | 0.571 | 0.618 | 204 | Available |
This table starts from the strongest GPT-labelled Book 3 coefficients and shows whether their direction survives manual-label and agreement-only refits.
| Feature | Baseline Direction | GPT labels on manual Book 3 scope | Manual labels only | Agreement-only labels | GPT labels, disagreements downweighted | Present | Sign Stable |
|---|---|---|---|---|---|---|---|
| εἰμί | mythic | +1.674 | +1.281 | +1.905 | +1.645 | 4 | yes |
| βασιλεύω | historical | -1.424 | -0.154 | -0.671 | -0.985 | 4 | yes |
| ἄλλοσ | historical | -1.098 | -0.326 | -0.577 | -1.073 | 4 | yes |
| εἰσ | historical | -0.993 | -1.087 | -0.800 | -0.972 | 4 | yes |
| ἀπό | mythic | +0.911 | +0.821 | +1.147 | +0.828 | 4 | yes |
| μή | historical | -0.869 | -1.075 | -0.892 | -0.908 | 4 | yes |
| εἰμί καί | mythic | +0.857 | +0.139 | +0.148 | +0.666 | 4 | yes |
| μάλιστα | historical | -0.856 | -0.896 | -0.699 | -0.870 | 4 | yes |
| σύν | historical | -0.838 | -0.496 | -0.502 | -0.682 | 4 | yes |
| δέ εἰμί | mythic | +0.819 | +0.336 | +0.619 | +0.674 | 4 | yes |
| ἡγέομαι | historical | -0.810 | -0.726 | -0.567 | -0.711 | 4 | yes |
| ποιέω δέ | mythic | +0.797 | +0.523 | +0.657 | +0.766 | 4 | yes |
| δέ γίγνομαι | mythic | +0.770 | +0.141 | -0.034 | +0.680 | 4 | no |
| εὑρίσκω | mythic | +0.752 | -0.144 | +0.475 | +0.551 | 4 | no |
| ἀπό οὗτοσ | mythic | +0.750 | +0.623 | +0.445 | +0.732 | 4 | yes |
| ὅτι | mythic | +0.745 | +0.707 | +0.878 | +0.697 | 4 | yes |
| βασιλεία | historical | -0.730 | -0.782 | -0.720 | -0.730 | 4 | yes |
| καί ἄλλοσ | historical | -0.729 | +0.150 | -0.302 | -0.639 | 4 | no |
| ἔνθα | mythic | +0.728 | +0.381 | +0.520 | 3 | yes | |
| τότε | historical | -0.706 | -0.439 | -0.633 | -0.542 | 4 | yes |
| ἔποσ | mythic | +0.703 | +0.319 | +0.689 | +0.683 | 4 | yes |
| καί εἰμί | mythic | +0.679 | -0.557 | -0.035 | +0.360 | 4 | no |
| πλησίον | mythic | +0.668 | +0.410 | +0.353 | +0.536 | 4 | yes |
| ὁμοῦ | historical | -0.667 | -0.714 | -0.419 | -0.477 | 4 | yes |
| πρίν | mythic | +0.667 | +0.259 | +0.292 | +0.525 | 4 | yes |
| καί καί | mythic | +0.662 | +0.778 | +0.827 | +0.694 | 4 | yes |
| ἐπί δέ | historical | -0.661 | -0.378 | +0.156 | -0.424 | 4 | no |
| πολύσ | historical | -0.647 | -0.497 | -0.699 | -0.400 | 4 | yes |
| οἶδα | mythic | +0.642 | +0.274 | +0.088 | +0.531 | 4 | yes |
| εἰσ οὗτοσ | mythic | +0.634 | +0.335 | +0.216 | +0.413 | 4 | yes |
This stress test resamples labels from the observed manual-vs-GPT confusion rates and refits the lemma model repeatedly. It is a sensitivity diagnostic, not a claim that either label set is ground truth.
| Feature | Present | Positive | Negative | Mean Coef | Min | Max | Sign Stability |
|---|---|---|---|---|---|---|---|
| εἰμί | 30 | 30 | 0 | +1.233 | +0.740 | +1.921 | 1.00 |
| βασιλεύω | 30 | 0 | 30 | -0.789 | -1.243 | -0.247 | 1.00 |
| ἄλλοσ | 30 | 1 | 29 | -0.569 | -1.388 | +0.112 | 0.97 |
| εἰσ | 30 | 1 | 29 | -0.908 | -1.605 | +0.220 | 0.97 |
| ἀπό | 30 | 29 | 1 | +0.672 | -0.015 | +1.436 | 0.97 |
| μή | 30 | 0 | 30 | -0.535 | -1.005 | -0.042 | 1.00 |
| εἰμί καί | 30 | 24 | 6 | +0.311 | -0.730 | +0.959 | 0.80 |
| μάλιστα | 30 | 1 | 29 | -0.464 | -1.025 | +0.421 | 0.97 |
| σύν | 30 | 1 | 29 | -0.469 | -1.150 | +0.059 | 0.97 |
| δέ εἰμί | 30 | 28 | 2 | +0.683 | -0.202 | +1.582 | 0.93 |
| ἡγέομαι | 27 | 1 | 26 | -0.470 | -0.945 | +0.090 | 0.96 |
| ποιέω δέ | 30 | 25 | 5 | +0.390 | -0.373 | +0.964 | 0.83 |
| δέ γίγνομαι | 21 | 13 | 8 | +0.063 | -0.381 | +0.667 | 0.62 |
| εὑρίσκω | 30 | 26 | 4 | +0.208 | -0.596 | +0.872 | 0.87 |
| ἀπό οὗτοσ | 29 | 27 | 2 | +0.408 | -0.081 | +0.776 | 0.93 |
| ὅτι | 30 | 27 | 3 | +0.332 | -0.123 | +0.886 | 0.90 |
| βασιλεία | 30 | 0 | 30 | -0.518 | -0.980 | -0.116 | 1.00 |
| καί ἄλλοσ | 30 | 2 | 28 | -0.459 | -1.055 | +0.124 | 0.93 |
| ἔνθα | 28 | 28 | 0 | +0.501 | +0.206 | +0.838 | 1.00 |
| τότε | 30 | 1 | 29 | -0.553 | -1.016 | +0.037 | 0.97 |
| ἔποσ | 24 | 23 | 1 | +0.470 | -0.013 | +0.684 | 0.96 |
| καί εἰμί | 30 | 22 | 8 | +0.266 | -0.450 | +0.897 | 0.73 |
| πλησίον | 29 | 20 | 9 | +0.221 | -0.301 | +0.921 | 0.69 |
| ὁμοῦ | 30 | 0 | 30 | -0.535 | -1.046 | -0.026 | 1.00 |
| πρίν | 29 | 28 | 1 | +0.385 | -0.014 | +0.643 | 0.97 |