Rendered at 12:11:46 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
augment_me 6 hours ago [-]
Its cool, but it goes against the whole optimization that NLP has gone through with BPE and redundancy/repetition in large data in general.
Put this on a coding task or English and you will underperform against tokenizer that exploit multi-char patterns and can decode much faster.
Given where things are going with spec decode (I've seen 32 token spec-decode pop up in the frontier models), you will need a very domain-specific use for this to be useful imo like massive tabular data, but at that point you already have everything structured, and might as well use an LLM with polars/sql/arrow.
yorwba 2 hours ago [-]
The whole point of this architecture is that it can also make use of multi-char patterns, but unlike BPE, they aren't hard-coded. And you can still use speculative decoding. Whether it works better or worse is something that you have to actually measure.
nivter 4 hours ago [-]
> you will need a very domain-specific use for this to be useful imo
The main premise of this idea is how general and widely applicable it is. They also show examples of how it works on images.
A common tokenizing paradigm would be a huge step towards consolidating how different modalities are preprocessed - something I believe is quite useful for intelligence to scale well across modalities.
vatsachak 12 hours ago [-]
Oh man, that haskell quicksort tokenization has got me feeling great
Put this on a coding task or English and you will underperform against tokenizer that exploit multi-char patterns and can decode much faster.
Given where things are going with spec decode (I've seen 32 token spec-decode pop up in the frontier models), you will need a very domain-specific use for this to be useful imo like massive tabular data, but at that point you already have everything structured, and might as well use an LLM with polars/sql/arrow.
The main premise of this idea is how general and widely applicable it is. They also show examples of how it works on images.
A common tokenizing paradigm would be a huge step towards consolidating how different modalities are preprocessed - something I believe is quite useful for intelligence to scale well across modalities.