Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
Vectors
classv2Vectors data is kept in the Vectors.data attribute, which should be an
instance of numpy.ndarray (for CPU vectors) or cupy.ndarray (for GPU
vectors). Multiple keys can be mapped to the same vector, and not all of the
rows in the table need to be assigned – so vectors.n_keys may be greater or
smaller than vectors.shape[0].
Vectors.__init__ method
Create a new vector store. You can set the vector values and keys directly on
initialization, or supply a shape keyword argument to create an empty table
you can add vectors to later.
| Name | Type | Description |
|---|---|---|
data | ndarray[ndim=1, dtype='float32'] | The vector data. |
keys | iterable | A sequence of keys aligned with the data. |
shape | tuple | Size of the table as (n_entries, n_columns), the number of entries and number of columns. Not required if you’re initializing the object with data and keys. |
name | unicode | A name to identify the vectors table. |
| RETURNS | Vectors | The newly created object. |
Vectors.__getitem__ method
Get a vector by key. If the key is not found in the table, a KeyError is
raised.
| Name | Type | Description |
|---|---|---|
key | int | The key to get the vector for. |
| returns | ndarray[ndim=1, dtype='float32'] | The vector for the key. |
Vectors.__setitem__ method
Set a vector for the given key.
| Name | Type | Description |
|---|---|---|
key | int | The key to set the vector for. |
vector | ndarray[ndim=1, dtype='float32'] | The vector to set. |
Vectors.__iter__ method
Iterate over the keys in the table.
| Name | Type | Description |
|---|---|---|
| YIELDS | int | A key in the table. |
Vectors.__len__ method
Return the number of vectors in the table.
| Name | Type | Description |
|---|---|---|
| RETURNS | int | The number of vectors in the table. |
Vectors.__contains__ method
Check whether a key has been mapped to a vector entry in the table.
| Name | Type | Description |
|---|---|---|
key | int | The key to check. |
| RETURNS | bool | Whether the key has a vector entry. |
Vectors.add method
Add a key to the table, optionally setting a vector value as well. Keys can be
mapped to an existing vector by setting row, or a new vector can be added.
When adding unicode keys, keep in mind that the Vectors class itself has no
StringStore, so you have to store the hash-to-string
mapping separately. If you need to manage the strings, you should use the
Vectors via the Vocab class, e.g. vocab.vectors.
| Name | Type | Description |
|---|---|---|
key | unicode / int | The key to add. |
vector | ndarray[ndim=1, dtype='float32'] | An optional vector to add for the key. |
row | int | An optional row number of a vector to map the key to. |
| RETURNS | int | The row the vector was added to. |
Vectors.resize method
Resize the underlying vectors array. If inplace=True, the memory is
reallocated. This may cause other references to the data to become invalid, so
only use inplace=True if you’re sure that’s what you want. If the number of
vectors is reduced, keys mapped to rows that have been deleted are removed.
These removed items are returned as a list of (key, row) tuples.
| Name | Type | Description |
|---|---|---|
shape | tuple | A (rows, dims) tuple describing the number of rows and dimensions. |
inplace | bool | Reallocate the memory. |
| RETURNS | list | The removed items as a list of (key, row) tuples. |
Vectors.keys method
A sequence of the keys in the table.
| Name | Type | Description |
|---|---|---|
| RETURNS | iterable | The keys. |
Vectors.values method
Iterate over vectors that have been assigned to at least one key. Note that some vectors may be unassigned, so the number of vectors returned may be less than the length of the vectors table.
| Name | Type | Description |
|---|---|---|
| YIELDS | ndarray[ndim=1, dtype='float32'] | A vector in the table. |
Vectors.items method
Iterate over (key, vector) pairs, in order.
| Name | Type | Description |
|---|---|---|
| YIELDS | tuple | (key, vector) pairs, in order. |
Vectors.find method
Look up one or more keys by row, or vice versa.
| Name | Type | Description |
|---|---|---|
key | unicode / int | Find the row that the given key points to. Returns int, -1 if missing. |
keys | iterable | Find rows that the keys point to. Returns ndarray. |
row | int | Find the first key that points to the row. Returns int. |
rows | iterable | Find the keys that point to the rows. Returns ndarray. |
| RETURNS | The requested key, keys, row or rows. |
Vectors.shape property
Get (rows, dims) tuples of number of rows and number of dimensions in the
vector table.
| Name | Type | Description |
|---|---|---|
| RETURNS | tuple | A (rows, dims) pair. |
Vectors.size property
The vector size, i.e. rows * dims.
| Name | Type | Description |
|---|---|---|
| RETURNS | int | The vector size. |
Vectors.is_full property
Whether the vectors table is full and has no slots are available for new keys.
If a table is full, it can be resized using
Vectors.resize.
| Name | Type | Description |
|---|---|---|
| RETURNS | bool | Whether the vectors table is full. |
Vectors.n_keys property
Get the number of keys in the table. Note that this is the number of all keys, not just unique vectors. If several keys are mapped are mapped to the same vectors, they will be counted individually.
| Name | Type | Description |
|---|---|---|
| RETURNS | int | The number of all keys in the table. |
Vectors.most_similar method
For each of the given vectors, find the n most similar entries to it, by
cosine. Queries are by vector. Results are returned as a
(keys, best_rows, scores) tuple. If queries is large, the calculations are
performed in chunks, to avoid consuming too much memory. You can set the
batch_size to control the size/space trade-off during the calculations.
| Name | Type | Description |
|---|---|---|
queries | ndarray | An array with one or more vectors. |
batch_size | int | The batch size to use. Default to 1024. |
n | int | The number of entries to return for each query. Defaults to 1. |
sort | bool | Whether to sort the entries returned by score. Defaults to True. |
| RETURNS | tuple | The most similar entries as a (keys, best_rows, scores) tuple. |
Vectors.to_disk method
Save the current state to a directory.
| Name | Type | Description |
|---|---|---|
path | unicode / Path | A path to a directory, which will be created if it doesn’t exist. Paths may be either strings or Path-like objects. |
Vectors.from_disk method
Loads state from a directory. Modifies the object in place and returns it.
| Name | Type | Description |
|---|---|---|
path | unicode / Path | A path to a directory. Paths may be either strings or Path-like objects. |
| RETURNS | Vectors | The modified Vectors object. |
Vectors.to_bytes method
Serialize the current state to a binary string.
| Name | Type | Description |
|---|---|---|
| RETURNS | bytes | The serialized form of the Vectors object. |
Vectors.from_bytes method
Load state from a binary string.
| Name | Type | Description |
|---|---|---|
data | bytes | The data to load from. |
| RETURNS | Vectors | The Vectors object. |
Attributes
| Name | Type | Description |
|---|---|---|
data | ndarray[ndim=1, dtype='float32'] | Stored vectors data. numpy is used for CPU vectors, cupy for GPU vectors. |
key2row | dict | Dictionary mapping word hashes to rows in the Vectors.data table. |
keys | ndarray[ndim=1, dtype='float32'] | Array keeping the keys in order, such that keys[vectors.key2row[key]] == key. |