Skip to content

Fix floret vectors StringStore sharing after Vocab unpickling (E018 with nlp.pipe n_process>1) - #13994

Open
joakimwar wants to merge 1 commit into
explosion:masterfrom
joakimwar:fix/floret-vectors-multiprocessing-unpickle
Open

joakimwar wants to merge 1 commit into
explosion:masterfrom
joakimwar:fix/floret-vectors-multiprocessing-unpickle

Conversation

@joakimwar

Copy link
Copy Markdown

Description

Fixes #13993: nlp.pipe(texts, n_process>1) immediately and deterministically raises [E018] (wrapped in [E871]) for any pipeline whose vocab.vectors.mode == "floret" (e.g. the official sv_core_news_lg), under the spawn multiprocessing start method (the default on macOS and Windows).

Cause: floret-mode vectors resolve each token hash back to its literal text via self.strings in Vectors.get_batch(). Normally vectors.strings is the same StringStore object as vocab.strings — the Vocab.vectors setter explicitly shares it. But unpickle_vocab assigned vocab.vectors = vectors before vocab.strings = sstore, so the setter shared the fresh Vocab()'s empty StringStore, which the very next line replaced. After unpickling — which is exactly how a spawned nlp.pipe worker receives the pipeline — the vectors table holds a reference to a discarded, empty store. Every string the tokenizer interns in the worker goes into vocab.strings, the floret lookup consults the orphaned store, and the first lookup raises E018.

Fix: assign vocab.strings before vocab.vectors in unpickle_vocab, so the existing setter logic shares the correct store. No new mechanism, just an ordering fix.

Testing (macOS arm64, Python 3.10, source build of master):

  • Verified the severed reference directly: after pickle.loads(pickle.dumps(nlp.vocab)), vocab.strings is vocab.vectors.strings is False before this fix and True after.
  • A/B against the issue's repro (1000 Swedish docs, sv_core_news_lg, batch_size=40, n_process=2): unpatched build fails with the exact E018/hash from the issue; patched build processes all 1000 docs.
  • spacy/tests/vocab_vectors and spacy/tests/serialize pass (222 passed).
  • Extended the existing test_pickle_vocab (which already uses floret-mode vectors) to assert that the unpickled vectors share the vocab's StringStore and that a string interned after unpickling resolves through vectors.strings. The new assertions fail on unpatched spaCy 3.8.7 and pass with this fix.

(🤖 Claude Code was used to investigate the bug and prepare this fix.)

Types of change

Bug fix.

Checklist

  • I confirm that I have the right to submit this contribution under the project's MIT license.
  • I ran the tests, and all new and existing tests passed.
  • My changes don't require a change to the documentation, or if they do, I've added all required information.
In unpickle_vocab, vocab.vectors was assigned while vocab.strings was
still the fresh Vocab's empty StringStore; assigning vocab.strings
afterwards left the vectors table holding a reference to the discarded
store. Strings interned after unpickling (e.g. by the tokenizer in an
nlp.pipe(n_process>1) spawn worker) were then missing from
vectors.strings, so floret-mode vectors raised E018 on string lookup in
Vectors.get_batch.

Set vocab.strings before vocab.vectors so the vectors setter shares the
vocab's StringStore with the vectors table, and extend test_pickle_vocab
to assert the sharing survives a pickle round-trip.

Fixes explosion#13993.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant