tokenizer

10 progetti condividono questo topic GitHub

Le linee collegano membri che sono misurabilmente correlati tra loro. La dimensione dei punti riflette le stelle.

🧬 Membri

LLMs-from-scratch

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

★ 100.1k

gigatoken

Language model tokenization at GB/s

★ 3.8k

tiktokenizer

Online playground for OpenAPI tokenizers

★ 1.7k

gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.).…

★ 827

Deepdive-llama3-from-scratch

Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement…

★ 632

UniTok

[NeurIPS 2025 Spotlight] A Unified Tokenizer for Visual Generation and Understanding

★ 529

tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken

★ 405

xcodec

AAAI 2025: Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model

★ 308

text2text

Text2Text Language Modeling Toolkit

★ 304

cevahir-ai

Full-stack open-source AI engine for building language models — tokenizer training, transformer…

★ 210

Misurato dai temi di GitHub condivisi da entrambi i progetti, ponderato in base a quanto è raro ciascun tema.