gpt-tokenizer

v2.5.1
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
BPE encoder decoder tokenizer GPT GPT-2 GPT-3 GPT-3.5 GPT-4 and 8 more...

Readme Versions Dependencies Scripts
2.5.1 2.5.0 2.4.1 2.4.0 2.3.0 2.2.3 2.2.2 2.2.1 2.2.0 2.1.2 2.1.1 2.1.0 2.0.0 2.0.0-beta.2 2.0.0-beta.1 1.0.5 1.0.4 1.0.3 1.0.2 1.0.1 1.0.0
npm i gpt-tokenizer

Metadata

  • MIT
  • Whatever
  • Bazyli Brzoska
  • released 10/21/2024

Downloads

Maintainers