src.corpora.indexer.IndexedDataset¶

class IndexedDataset(cache_dir, seq_len: int, stride: Optional[int] = None)[source]¶

Bases: IterDataPipe[BatchEncoding]

Methods

`build_or_load`
`register_datapipe_as_function`
`register_function`
`set_getstate_hook`
`set_reduce_ex_hook`

Attributes