A model architecture is a function that wires up a Model instance, which you can then use in a pipeline component or as a layer of a larger network. This page documents spaCy’s built-in architectures that are used for different NLP tasks. All trainable built-in components expect a model argument defined in the config and document their the default architecture. Custom architectures can be registered using the @spacy.registry.architectures decorator and used as part of the training config. Also see the usage documentation on layers and model architectures.

Tok2Vec architectures

Source

spacy.Tok2Vec.v2

Construct a tok2vec model out of two subnetworks: one for embedding and one for encoding. See the “Embed, Encode, Attend, Predict” blog post for background.

spacy.HashEmbedCNN.v2

Build spaCy’s “standard” tok2vec layer. This layer is defined by a MultiHashEmbed embedding layer that uses subword features, and a MaxoutWindowEncoder encoding layer consisting of a CNN and a layer-normalized maxout activation function.

spacy.Tok2VecListener.v1

A listener is used as a sublayer within a component such as a DependencyParser, EntityRecognizeror TextCategorizer. Usually you’ll have multiple listeners connecting to a single upstream Tok2Vec component that’s earlier in the pipeline. The listener layers act as proxies, passing the predictions from the Tok2Vec component into downstream components, and communicating gradients back upstream.

Instead of defining its own Tok2Vec instance, a model architecture like Tagger can define a listener as its tok2vec argument that connects to the shared tok2vec component in the pipeline.

Listeners work by caching the Tok2Vec output for a given batch of Docs. This means that in order for a component to work with the listener, the batch of Docs passed to the listener must be the same as the batch of Docs passed to the Tok2Vec. As a result, any manipulation of the Docs which would affect Tok2Vec output, such as to create special contexts or remove Docs for which no prediction can be made, must happen inside the model, after the call to the Tok2Vec component.

spacy.MultiHashEmbed.v2

Construct an embedding layer that separately embeds a number of lexical attributes using hash embedding, concatenates the results, and passes it through a feed-forward subnetwork to build a mixed representation. The features used can be configured with the attrs argument. The suggested attributes are NORM, PREFIX, SUFFIX and SHAPE. This lets the model take into account some subword information, without construction a fully character-based representation. If pretrained vectors are available, they can be included in the representation as well, with the vectors table kept static (i.e. it’s not updated).

spacy.CharacterEmbed.v2

Construct an embedded representation based on character embeddings, using a feed-forward network. A fixed number of UTF-8 byte characters are used for each word, taken from the beginning and end of the word equally. Padding is used in the center for words that are too short.

For instance, let’s say nC=4, and the word is “jumping”. The characters used will be "jung" (two from the start, two from the end). If we had nC=8, the characters would be "jumpping": 4 from the start, 4 from the end. This ensures that the final character is always in the last position, instead of being in an arbitrary position depending on the word length.

The characters are embedded in a embedding table with a given number of rows, and the vectors concatenated. A hash-embedded vector of the NORM of the word is also concatenated on, and the result is then passed through a feed-forward network to construct a single vector to represent the information.

spacy.MaxoutWindowEncoder.v2

Encode context using convolutions with maxout activation, layer normalization and residual connections.

spacy.MishWindowEncoder.v2

Encode context using convolutions with Mish activation, layer normalization and residual connections.

spacy.TorchBiLSTMEncoder.v1

Encode context using bidirectional LSTM layers. Requires PyTorch.

spacy.StaticVectors.v2

Embed Doc objects with their vocab’s vectors table, applying a learned linear projection to control the dimensionality. Unknown tokens are mapped to a zero vector. See the documentation on static vectors for details.

spacy.FeatureExtractor.v1

Extract arrays of input features from Doc objects. Expects a list of feature names to extract, which should refer to token attributes.

Transformer architectures

Source

The following architectures are provided by the package spacy-transformers. See the usage documentation for how to integrate the architectures into your training config.

spacy-transformers.TransformerModel.v3

Load and wrap a transformer model from the HuggingFace transformers library. You can use any transformer that has pretrained weights and a PyTorch implementation. The name variable is passed through to the underlying library, so it can be either a string or a path. If it’s a string, the pretrained weights will be downloaded via the transformers library if they are not already available locally.

In order to support longer documents, the TransformerModel layer allows you to pass in a get_spans function that will divide up the Doc objects before passing them through the transformer. Your spans are allowed to overlap or exclude tokens. This layer is usually used directly by the Transformer component, which allows you to share the transformer weights across your pipeline. For a layer that’s configured for use in other components, see Tok2VecTransformer.

spacy-transformers.TransformerListener.v1

Create a TransformerListener layer, which will connect to a Transformer component earlier in the pipeline. The layer takes a list of Doc objects as input, and produces a list of 2-dimensional arrays as output, with each array having one row per token. Most spaCy models expect a sublayer with this signature, making it easy to connect them to a transformer model via this sublayer. Transformer models usually operate over wordpieces, which usually don’t align one-to-one against spaCy tokens. The layer therefore requires a reduction operation in order to calculate a single token vector given zero or more wordpiece vectors.

spacy-transformers.Tok2VecTransformer.v3

Use a transformer as a Tok2Vec layer directly. This does not allow multiple components to share the transformer weights and does not allow the transformer to set annotations into the Doc object, but it’s a simpler solution if you only need the transformer within one component.

Curated Transformer architectures

Source

The following architectures are provided by the package spacy-curated-transformers. See the usage documentation for how to integrate the architectures into your training config.

When loading the model from the Hugging Face Hub, the model config’s parameters must be same as the hyperparameters used by the pre-trained model. The init fill-curated-transformer CLI command can be used to automatically fill in these values.

spacy-curated-transformers.AlbertTransformer.v1

Construct an ALBERT transformer model.

spacy-curated-transformers.BertTransformer.v1

Construct a BERT transformer model.

spacy-curated-transformers.CamembertTransformer.v1

Construct a CamemBERT transformer model.

spacy-curated-transformers.RobertaTransformer.v1

Construct a RoBERTa transformer model.

spacy-curated-transformers.XlmrTransformer.v1

Construct a XLM-RoBERTa transformer model.

spacy-curated-transformers.ScalarWeight.v1

Construct a model that accepts a list of transformer layer outputs and returns a weighted representation of the same.

spacy-curated-transformers.TransformerLayersListener.v1

Construct a listener layer that communicates with one or more upstream Transformer components. This layer extracts the output of the last transformer layer and performs pooling over the individual pieces of each Doc token, returning their corresponding representations. The upstream name should either be the wildcard string ’*’, or the name of the Transformer component.

In almost all cases, the wildcard string will suffice as there’ll only be one upstream Transformer component. But in certain situations, e.g: you have disjoint datasets for certain tasks, or you’d like to use a pre-trained pipeline but a downstream task requires its own token representations, you could end up with more than one Transformer component in the pipeline.

spacy-curated-transformers.LastTransformerLayerListener.v1

Construct a listener layer that communicates with one or more upstream Transformer components. This layer extracts the output of the last transformer layer and performs pooling over the individual pieces of each Doc token, returning their corresponding representations. The upstream name should either be the wildcard string ’*’, or the name of the Transformer component.

In almost all cases, the wildcard string will suffice as there’ll only be one upstream Transformer component. But in certain situations, e.g: you have disjoint datasets for certain tasks, or you’d like to use a pre-trained pipeline but a downstream task requires its own token representations, you could end up with more than one Transformer component in the pipeline.

spacy-curated-transformers.ScalarWeightingListener.v1

Construct a listener layer that communicates with one or more upstream Transformer components. This layer calculates a weighted representation of all transformer layer outputs and performs pooling over the individual pieces of each Doc token, returning their corresponding representations.

Requires its upstream Transformer components to return all layer outputs from their models. The upstream name should either be the wildcard string ’*’, or the name of the Transformer component.

In almost all cases, the wildcard string will suffice as there’ll only be one upstream Transformer component. But in certain situations, e.g: you have disjoint datasets for certain tasks, or you’d like to use a pre-trained pipeline but a downstream task requires its own token representations, you could end up with more than one Transformer component in the pipeline.

spacy-curated-transformers.BertWordpieceEncoder.v1

Construct a WordPiece piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers. This encoder also splits each token on punctuation characters, as expected by most BERT models.

This model must be separately initialized using an appropriate loader.

spacy-curated-transformers.ByteBpeEncoder.v1

Construct a Byte-BPE piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers.

This model must be separately initialized using an appropriate loader.

spacy-curated-transformers.CamembertSentencepieceEncoder.v1

Construct a SentencePiece piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers with CamemBERT post-processing applied.

This model must be separately initialized using an appropriate loader.

spacy-curated-transformers.CharEncoder.v1

Construct a character piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers.

This model must be separately initialized using an appropriate loader.

spacy-curated-transformers.SentencepieceEncoder.v1

Construct a SentencePiece piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers.

This model must be separately initialized using an appropriate loader.

spacy-curated-transformers.WordpieceEncoder.v1

Construct a WordPiece piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers. This encoder also splits each token on punctuation characters, as expected by most BERT models.

This model must be separately initialized using an appropriate loader.

spacy-curated-transformers.XlmrSentencepieceEncoder.v1

Construct a SentencePiece piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers with XLM-RoBERTa post-processing applied.

This model must be separately initialized using an appropriate loader.

Pretraining architectures

Source

The spacy pretrain command lets you initialize a Tok2Vec layer in your pipeline with information from raw text. To this end, additional layers are added to build a network for a temporary task that forces the Tok2Vec layer to learn something about sentence structure and word cooccurrence statistics. Two pretraining objectives are available, both of which are variants of the cloze task Devlin et al. (2018) introduced for BERT.

For more information, see the section on pretraining.

spacy.PretrainVectors.v1

Predict the word’s vector from a static embeddings table as pretraining objective for a Tok2Vec layer. To use this objective, make sure that the initialize.vectors section in the config refers to a model with static vectors.

spacy.PretrainCharacters.v1

Predict some number of leading and trailing UTF-8 bytes as pretraining objective for a Tok2Vec layer.

Parser & NER architectures

spacy.TransitionBasedParser.v2

Source

Build a transition-based parser model. Can apply to NER or dependency parsing. Transition-based parsing is an approach to structured prediction where the task of predicting the structure is mapped to a series of state transitions. You might find this tutorial helpful for background information. The neural network state prediction model consists of either two or three subnetworks:

• tok2vec: Map each token into a vector representation. This subnetwork is run once for each batch.
• lower: Construct a feature-specific vector for each (token, feature) pair. This is also run once for each batch. Constructing the state representation is then a matter of summing the component features and applying the non-linearity.
• upper (optional): A feed-forward network that predicts scores from the state representation. If not present, the output from the lower model is used as action scores directly.

Tagging architectures

Source

spacy.Tagger.v2

Build a tagger model, using a provided token-to-vector component. The tagger model adds a linear layer with softmax activation to predict scores given the token vectors.

Text classification architectures

Source

A text classification architecture needs to take a Doc as input, and produce a score for each potential label class. Textcat challenges can be binary (e.g. sentiment analysis) or involve multiple possible labels. Multi-label challenges can either have mutually exclusive labels (each example has exactly one label), or multiple labels may be applicable at the same time.

As the properties of text classification problems can vary widely, we provide several different built-in architectures. It is recommended to experiment with different architectures and settings to determine what works best on your specific data and challenge.

spacy.TextCatEnsemble.v2

Stacked ensemble of a linear bag-of-words model and a neural network model. The neural network is built upon a Tok2Vec layer and uses attention. The setting for whether or not this model should cater for multi-label classification, is taken from the linear model, where it is stored in model.attrs["multi_label"].

spacy.TextCatBOW.v3

An n-gram “bag-of-words” model. This architecture should run much faster than the others, but may not be as accurate, especially if texts are short.

spacy.TextCatParametricAttention.v1

A neural network model that is built upon Tok2Vec and uses parametric attention to attend to tokens that are relevant to text classification.

spacy.TextCatReduce.v1

A classifier that pools token hidden representations of each Doc using first, max or mean reduction and then applies a classification layer. Reductions are concatenated when multiple reductions are used.

Span classification architectures

Source

spacy.SpanCategorizer.v1

Build a span categorizer model to power a SpanCategorizer component, given a token-to-vector model, a reducer model to map the sequence of vectors for each span down to a single vector, and a scorer model to map the vectors to probabilities.

spacy.mean_max_reducer.v1

Reduce sequences by concatenating their mean and max pooled vectors, and then combine the concatenated vectors with a hidden layer.

Entity linking architectures

Source

An EntityLinker component disambiguates textual mentions (tagged as named entities) to unique identifiers, grounding the named entities into the “real world”. This requires 3 main components:

• A KnowledgeBase (KB) holding the unique identifiers, potential synonyms and prior probabilities.
• A candidate generation step to produce a set of likely identifiers, given a certain textual mention.
• A machine learning Model that picks the most plausible ID from the set of candidates.

spacy.EntityLinker.v2

The EntityLinker model architecture is a Thinc Model with a Linear output layer.

spacy.EmptyKB.v1

A function that creates an empty KnowledgeBase from a Vocab instance.

spacy.EmptyKB.v2

A function that creates an empty KnowledgeBase from a Vocab instance. This is the default when a new entity linker component is created. It returns a Callable[[Vocab, int], InMemoryLookupKB].

spacy.KBFromFile.v1

A function that reads an existing KnowledgeBase from file.

spacy.CandidateGenerator.v1

A function that takes as input a KnowledgeBase and a Span object denoting a named entity, and returns a list of plausible Candidate objects. The default CandidateGenerator uses the text of a mention to find its potential aliases in the KnowledgeBase. Note that this function is case-dependent.

spacy.CandidateBatchGenerator.v1

A function that takes as input a KnowledgeBase and an Iterable of Span objects denoting named entities, and returns a list of plausible Candidate objects per specified Span. The default CandidateBatchGenerator uses the text of a mention to find its potential aliases in the KnowledgeBase. Note that this function is case-dependent.

Coreference experimental

A CoreferenceResolver component identifies tokens that refer to the same entity. A SpanResolver component infers spans from single tokens. Together these components can be used to reproduce traditional coreference models. You can also omit the SpanResolver if working with only token-level clusters is acceptable.

spacy-experimental.Coref.v1 experimental

The Coref model architecture is a Thinc Model.

spacy-experimental.SpanResolver.v1 experimental

The SpanResolver model architecture is a Thinc Model. Note that MentionClusters is List[List[Tuple[int, int]]].