Pipeline Functions · spaCy API Documentation

merge_noun_chunks function

Merge noun chunks into a single token. Also available via the string name "merge_noun_chunks". After initialization, the component is typically added to the processing pipeline using nlp.add_pipe.

Example

texts = [t.text for t in nlp("I have a blue car")]
assert texts == ["I", "have", "a", "blue", "car"]

merge_nps = nlp.create_pipe("merge_noun_chunks")
nlp.add_pipe(merge_nps)

texts = [t.text for t in nlp("I have a blue car")]
assert texts == ["I", "have", "a blue car"]

Name	Type	Description
`doc`	`Doc`	The `Doc` object to process, e.g. the `Doc` in the pipeline.
RETURNS	`Doc`	The modified `Doc` with merged noun chunks.

merge_entities function

Merge named entities into a single token. Also available via the string name "merge_entities". After initialization, the component is typically added to the processing pipeline using nlp.add_pipe.

Example

texts = [t.text for t in nlp("I like David Bowie")]
assert texts == ["I", "like", "David", "Bowie"]

merge_ents = nlp.create_pipe("merge_entities")
nlp.add_pipe(merge_ents)

texts = [t.text for t in nlp("I like David Bowie")]
assert texts == ["I", "like", "David Bowie"]

Name	Type	Description
`doc`	`Doc`	The `Doc` object to process, e.g. the `Doc` in the pipeline.
RETURNS	`Doc`	The modified `Doc` with merged entities.

merge_subtokens functionv2.1

Merge subtokens into a single token. Also available via the string name "merge_subtokens". After initialization, the component is typically added to the processing pipeline using nlp.add_pipe.

As of v2.1, the parser is able to predict “subtokens” that should be merged into one single token later on. This is especially relevant for languages like Chinese, Japanese or Korean, where a “word” isn’t defined as a whitespace-delimited sequence of characters. Under the hood, this component uses the Matcher to find sequences of tokens with the dependency label "subtok" and then merges them into a single token.

Example

Note that this example assumes a custom Chinese model that oversegments and was trained to predict subtokens.

doc = nlp("拜托")
print([(token.text, token.dep_) for token in doc])
# [('拜', 'subtok'), ('托', 'subtok')]

merge_subtok = nlp.create_pipe("merge_subtokens")
nlp.add_pipe(merge_subtok)

doc = nlp("拜托")
print([token.text for token in doc])
# ['拜托']

Name	Type	Description
`doc`	`Doc`	The `Doc` object to process, e.g. the `Doc` in the pipeline.
`label`	unicode	The subtoken dependency label. Defaults to `"subtok"`.
RETURNS	`Doc`	The modified `Doc` with merged subtokens.

Dec	JAN	Feb
	06
2019	2020	2021