Elisabeth Fittschen, Sabrina Xin Li, Tom Lippincott, Leshem Chosem and Craig Messner. ACL Anthology. Findings of EACL 2026.

We use compute and data efficient methods to pretrain a battery of historically-specific models over a relatively limited budget of tokens. We show that this approach leaks far less temporally-inappropriate information than finetuning an existing LLM while retaining adequate performance to do lexical change detection.

Our approach can be used for any corpus with delineations between collections of works. Code is available on GitHub and models and data on HuggingFace.