All pull requests

Pull requests

• Allow huggingface_hub 2.0

  #2486·musicinmybrain opened Oct 1, 2026Contributor·

  ·

• fix(added-vocab): keep the original content of normalized added tokens

  #2485·MohammadHijjawi97 opened Oct 1, 2026·

  ·
  2

• fix(bpe): apply end_of_word_suffix to single-character words

  #2484·MohammadHijjawi97 opened Oct 1, 2026·

  ·
  2

• docs: using the crate and its features

  #2481·SBrandeis opened Sep 30, 2026Collaborator·11/11·

  ·
  1

• chore(refactor): move progress bars from tk-encode to tk-train

  #2480·SBrandeis opened Sep 30, 2026Collaborator·10/11·

  ·
  1

• feat(umbrella): expose tk-train through a train feature

  #2479·SBrandeis opened Sep 30, 2026Collaborator·9/11·

  ·
  1

• chore(bindings): use the umbrella tokenizers crate instead of the tk-* sub crates

  #2478·SBrandeis opened Sep 30, 2026Collaborator·8/11·

  ·
  1

• feat(umbrella): rework umbrella crate feature set

  #2477·SBrandeis opened Sep 30, 2026Collaborator·7/11·

  ·
  1

• chore(dx): rework tk-train feature set

  #2476·SBrandeis opened Sep 30, 2026Collaborator·6/11·

  ·
  1

• chore(dx): add tk-train to the cargo workspace

  #2475·SBrandeis opened Sep 30, 2026Collaborator·5/11·

  ·
  1

• fix(tk-train): ParityBpeTrainer: train into Vocab and Merges

  #2474·SBrandeis opened Sep 30, 2026Collaborator·4/11·

  ·
  1

• chore(tk-train): fmt, remove dead code, stale docs, and unused tk-convert dependency

  #2473·SBrandeis opened Sep 30, 2026Collaborator·3/11·

  ·
  1

• fix(tk-encode): remove unicode-normalization-alignments dependency

  #2472·SBrandeis opened Sep 30, 2026Collaborator·2/11·

  ·
  1

• chore(tk-encode): drop itertools, paste and unstable_wasm

  #2471·SBrandeis opened Sep 30, 2026Collaborator·1/11·

  ·
  1

• feat(encode): honour truncation stride as Encoding::overflowing, derive masks from a layout

  #2468·ysjprojects opened Sep 29, 2026·

  ·

• fix(BucketVocabStore): Explicit duplicate handling

  #2467·SBrandeis opened Sep 28, 2026Collaborator·

  ·
  1

• fix(byte-level): Support sparse byte atoms

  #2466·SBrandeis opened Sep 28, 2026Collaborator·

  ·
  2

• chore(bitcannon): drop unused dependency, lint

  #2465·SBrandeis opened Sep 28, 2026Collaborator·

  ·
  2

• fix(python): xxx_array getters copy the underlying data

  #2464·SBrandeis opened Sep 28, 2026Collaborator·

  ·
  1

  3

• chore(deps): bump the actions group with 4 updates

  dependenciesgithub_actions
  #2463·dependabot[bot] opened Sep 27, 2026Bot·

  ·
  1

• Support ByteLevel add_prefix_space during canonicalization

  #2462·ducanhnguyen223 opened Sep 27, 2026·

  ·
  1

  1

• Load byte-fallback BPE vocabs that lack some <0xNN> codes

  #2461·daaain opened Sep 26, 2026·

  ·
  2

• fix: make tk-train compile and add a CI workflow

  #2457·SBrandeis opened Sep 25, 2026Collaborator·

  ·
  1

• chore(cleanup): Remove dead code, rename, simplify

  #2455·SBrandeis opened Sep 24, 2026Collaborator·

  ·
  1

• feat(python): role_to_token

  #2454·SBrandeis opened Sep 24, 2026Collaborator·

  ·
  1