cohere-ai/tokenizer
Go
Captured source
source ↗published Jan 3, 2022seen 6dcaptured 8hhttp 200method plain
cohere-ai/tokenizer
Description: BPE tokenization implemented in Golang 💙
Language: Go
License: Apache-2.0
Stars: 12
Forks: 2
Open issues: 7
Created: 2022-01-03T01:15:49Z
Pushed: 2023-10-02T13:59:32Z
Default branch: main
Fork: no
Archived: yes
README:
Outdated
Please refer to tokenize and detokenize APIs for up to date Cohere tokenizers.