Does the vocabulary model generalise across books?
Each row trains the same lemma/proper-noun-removed model on all other books and tests on the held-out book. Books 4 and 8 are included here deliberately as stress tests.
| Book | Test | Hist Test | Myth Test | Hist Train | Myth Train | Accuracy | Baseline | Delta | Hist F1 | Myth F1 | HH/HM/MH/MM |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 938 | 481 | 457 | 2,953 | 3,141 | 0.694 | 0.487 | +0.207 | 0.690 | 0.698 | 319/162/125/332 |
| 2 | 754 | 176 | 578 | 3,258 | 3,020 | 0.731 | 0.233 | +0.497 | 0.550 | 0.808 | 124/52/151/427 |
| 3 | 623 | 315 | 308 | 3,119 | 3,290 | 0.709 | 0.494 | +0.215 | 0.690 | 0.727 | 201/114/67/241 |
| 4 | 796 | 540 | 256 | 2,894 | 3,342 | 0.724 | 0.322 | +0.402 | 0.784 | 0.615 | 400/140/80/176 |
| 5 | 500 | 215 | 285 | 3,219 | 3,313 | 0.648 | 0.570 | +0.078 | 0.600 | 0.686 | 132/83/93/192 |
| 6 | 484 | 347 | 137 | 3,087 | 3,461 | 0.657 | 0.283 | +0.374 | 0.737 | 0.506 | 233/114/52/85 |
| 7 | 614 | 395 | 219 | 3,039 | 3,379 | 0.735 | 0.357 | +0.378 | 0.793 | 0.630 | 312/83/80/139 |
| 8 | 812 | 337 | 475 | 3,097 | 3,123 | 0.717 | 0.585 | +0.132 | 0.664 | 0.755 | 227/110/120/355 |
| 9 | 713 | 234 | 479 | 3,200 | 3,119 | 0.745 | 0.328 | +0.417 | 0.646 | 0.800 | 166/68/114/365 |
| 10 | 798 | 394 | 404 | 3,040 | 3,194 | 0.736 | 0.506 | +0.229 | 0.744 | 0.727 | 306/88/123/281 |