Homepage

tokenizers

Licenses: Apache-2.0
Published: 6 years ago Last updated: 3 months ago Latest version: 0.23.1 Latest non-vulnerable version: 0.23.1
pip
GitHub

Package Health Score

90/100
  • securityNo known security issues
  • popularityKey ecosystem project
  • maintenanceSustainable
  • communityActive

Maintenance

SUSTAINABLE

Commit Frequency

Loading chart...

Open Issues
116
Open PR
128
New PRS
1
Last Release
3 months ago
Last Commit
23 days ago
Maintainers
150

Further analysis of the maintenance status of tokenizers based on released PyPI versions cadence, the repository activity, and other data points determined that its maintenance is Sustainable.

We found that tokenizers demonstrates a positive version release cadence with at least one new version released in the past 12 months.

As a healthy sign for on-going project maintenance, we found that the GitHub repository had at least 1 pull request or issue interacted with by the community.

Popularity

KEY ECOSYSTEM PROJECT
GitHub Stars
10.9k
Forks
1.2k
Contributors
150

Based on project statistics from the GitHub repository for the PyPI package tokenizers, we found that it has been starred 10,898 times.

Community

ACTIVE
Readme
Yes
Contributing.md
Yes
Code of Conduct
No
Contributors
150
Funding
No
LICENSE
Apache-2.0

A good and healthy external contribution signal for tokenizers project, which invites more than one hundred open source maintainers to collaborate on the repository.

We found a way for you to contribute to the project! Looks like tokenizers is missing a Code of Conduct.

License

Apache-2.0[0,);
Report a new vulnerability Found a mistake?

Direct Vulnerabilities

No direct vulnerabilities have been found for this package in Snyk’s vulnerability database. This does not include vulnerabilities belonging to this package’s dependencies.

Does your project rely on vulnerable package dependencies?

Automatically find and fix vulnerabilities affecting your projects. Snyk scans for vulnerabilities (in both your packages & their dependencies) and provides automated fixes for free.

Scan for indirect vulnerabilities

Security

NO KNOWN SECURITY ISSUES
Show only direct vulnerabilities in latest version
No vulnerabilities found

No vulnerabilities found in the latest version

Package versions

100 VERSIONS IN TOTAL See all versions
versionpublisheddirect vulnerabilities
0.23.127 Apr, 2026
  • 0
    C
  • 0
    H
  • 0
    M
  • 0
    L
0.23.1rc027 Apr, 2026
  • 0
    C
  • 0
    H
  • 0
    M
  • 0
    L
0.23.0rc024 Apr, 2026
  • 0
    C
  • 0
    H
  • 0
    M
  • 0
    L
0.22.25 Jan, 2026
  • 0
    C
  • 0
    H
  • 0
    M
  • 0
    L
0.22.2rc02 Dec, 2025
  • 0
    C
  • 0
    H
  • 0
    M
  • 0
    L
0.22.119 Sep, 2025
  • 0
    C
  • 0
    H
  • 0
    M
  • 0
    L
0.22.1rc019 Sep, 2025
  • 0
    C
  • 0
    H
  • 0
    M
  • 0
    L
0.22.029 Aug, 2025
  • 0
    C
  • 0
    H
  • 0
    M
  • 0
    L
0.22.0rc029 Aug, 2025
  • 0
    C
  • 0
    H
  • 0
    M
  • 0
    L
0.21.428 Jul, 2025
  • 0
    C
  • 0
    H
  • 0
    M
  • 0
    L