Lingwei Gu, Nour Jedidi, Jimmy Lin

Researchers have released NanoKnow, a benchmark dataset that maps questions from Natural Questions and SQuAD to whether their answers appear in nanochat's fully transparent pre-training corpus. This enables direct measurement of how LLMs encode and rely on parametric knowledge versus external evidence. Experiments across eight nanochat checkpoints reveal that answer frequency in training data strongly influences closed-book accuracy, that external evidence can reduce this dependence but remains complementary to parametric knowledge, and that irrelevant context actively harms performance based on position and volume.