AI21Labs/ai21-tokenizer v1.0.0

v1.0.0

Repository: AI21Labs/ai21-tokenizer

Tag: v1.0.0

Published: 2025-05-28T14:35:49Z

Prerelease: no

Release notes:

v1.0.0 (2025-05-28)

Bug Fixes

• Load from huggingface if cache fails (#105, `4b78e7b`)

• Paths to huggingface (#105, `4b78e7b`)

Continuous Integration

• Add hf token from secrets for tests (#100, `d5eeb96`)

• Change env (#101, `499ff81`)

• Fix env var in github workflow (#101, `499ff81`)

• Updated github actions (#105, `4b78e7b`)

Documentation

• Add note about jamba 1.5s (#100, `d5eeb96`)

---

Detailed Changes: v0.12.0...v1.0.0