tokenization

Unsupervised text tokenizer focused on computational efficiency

Fast and customizable text tokenization library with BPE and SentencePiece support

The Libre Multilingual Analyzer, a Natural Language Processing (NLP) C++ toolkit.

C++ Library for tokenizing MIDI files, designed to be compatible with the MIDITok python library

Smart Language Model

[NeurIPS 2025] Code for BEAST Experiments on CALVIN and LIBERO.

A lightweight, object-oriented programming language with integrated web server capabilities, designed for rapid prototyping, educational purposes, and backend development.

Tokenizers and Machine Learning Models for biological sequence data

R package for Byte Pair Encoding based on YouTokenToMe

This project is a fully functional compiler for the TINY programming language, which is a language that supports basic arithmetic, boolean, and control flow operations. The compiler can scan, parse, and run code written in the TINY language.

A custom tokenizer (byte level BPE) trained by me to try to replicate LLAMA 2's massive token vocabulary

Lightweight, header-only Byte Pair Encoding (BPE) trainer in modern C++17. Produces HuggingFace-compatible vocabularies for transformers and integrates with Modern Text Tokenizer.

Designing a compiler for a defined language

Byte level BPE are algorithm for sub word level tokenization.

Code example for the Fundamentals of Programming course at the University of Guilan, Fall 2016.

Runtime lexers for real languages, and exact decisions about where a maximal-munch scan may begin: certified boundaries, parallel scanning, and an auditor for flex, re2c, ANTLR and logos scanners.

Implementation of C++ lexical analyzer to demonstrate how it actually works as a part of the compiler.

Statically‑typed, block‑structured programming language

tf-idf (Term-Frequency Inverse-Document-Frequency) C++ Library