The delimiter-fidelity probe uses synthetic blocks and demonstrates that sanitizers destroy delimiter-bearing content, but does not estimate how often such content occurs in a real workload. The tokenizer audit includes only repositories that ship a chat template and load without executing repository code, so models distributed with custom tokenizer code are under-covered. · CiteArk
The delimiter-fidelity probe uses synthetic blocks and demonstrates that sanitizers destroy delimiter-bearing content, but does not estimate how often such content occurs in a real workload. The tokenizer audit includes only repositories that ship a chat template and load without executing repository code, so models distributed with custom tokenizer code are under-covered.
来源：fixed-paper:PDF p. 5, Limitations, third paragraph
