Pausanias Analysis

Does the vocabulary model generalise across books?

Book-Held-Out Robustness

Each row trains the same lemma/proper-noun-removed model on all other books and tests on the held-out book. Books 4 and 8 are included here deliberately as stress tests.

0.713weighted accuracy
0.709mean book accuracy
7,032held-out sentences
BookTestHist TestMyth TestHist TrainMyth TrainAccuracyBaselineDeltaHist F1Myth F1HH/HM/MH/MM
1 938 481 457 2,953 3,141 0.694 0.487 +0.207 0.690 0.698 319/162/125/332
2 754 176 578 3,258 3,020 0.731 0.233 +0.497 0.550 0.808 124/52/151/427
3 623 315 308 3,119 3,290 0.709 0.494 +0.215 0.690 0.727 201/114/67/241
4 796 540 256 2,894 3,342 0.724 0.322 +0.402 0.784 0.615 400/140/80/176
5 500 215 285 3,219 3,313 0.648 0.570 +0.078 0.600 0.686 132/83/93/192
6 484 347 137 3,087 3,461 0.657 0.283 +0.374 0.737 0.506 233/114/52/85
7 614 395 219 3,039 3,379 0.735 0.357 +0.378 0.793 0.630 312/83/80/139
8 812 337 475 3,097 3,123 0.717 0.585 +0.132 0.664 0.755 227/110/120/355
9 713 234 479 3,200 3,119 0.745 0.328 +0.417 0.646 0.800 166/68/114/365
10 798 394 404 3,040 3,194 0.736 0.506 +0.229 0.744 0.727 306/88/123/281