@mlc-ai/web-tokenizers
v0.1.5 machine_learning llm nlp
WebTokenizers
| NPM Package | WebLLM |
WebTokenizers is a javascript binding library that can be universally deployed. It wraps and binds the HuggingFace tokenizers library and sentencepiece and provides a minimum common interface in havascript.
Build from Srouce
source /path/to/emsdk_env.sh
npm install
npm run build
To try out the test webpage
cd tests
npm start
npm i @mlc-ai/web-tokenizers
Metadata
- Apache-2.0
- Whatever
- Unknown
- released 8/12/2024