src.corpora.detokenization

detokenization.py

Handle detokenization for different dataset for zero-shot LM evaluation.

Functions

wikitext_detokenize

Wikitext is whitespace tokenized and we remove these whitespaces.