learned-tokenization

Here are 2 public repositories matching this topic...

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

My attempts at applying Soundstream design on learned tokenization of text and then applying hierarchical attention to text generation

To associate your repository with the learned-tokenization topic, visit your repo's landing page and select "manage topics."