0
$\begingroup$

Positional embeddings are introduced into a transformer in order to add positional information to a word embedding.

Now, suppose we have an existing data embedding that can be for any data domain word/image. We don't have the original text/image before being encoded but the final embedding. Is it possible to extract positional embedding from an embedding or do we have to run positional embedding on the raw input (text/image) please? Is there a way around it in case we don't have the original raw input but only its embedding?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Not possible in a typical Transformer AFAIK. Most neural networks are non-invertible mappings, so you cannot guaranteed reconstruct their inputs from their hidden layers. For transformers specifically, the positional embedding is added to the token embedding before being fed into self-attention, so there not an easy way to disentangle these that I'm aware of.

Improve this answer
$\endgroup$
1
$\begingroup$

In the Transformer architecture, the positional embedding is added to the data embedding vectors so, if you had just the embedded data and kept the ordering between data vectors, you could just pass it through the normal transformer:

enter image description here

Improve this answer
$\endgroup$
5
  • $\begingroup$ No the data embedded before was not aware of the position. So I guess I can not add positional embedding without the raw data then? But I have something interesting though in the embedded data, the data has a groups of embeddings for per cluster (group of greetings sentences), (group of business sentences), ... etc. $[embed_1, embed_2, ..., embed_n]$ (greetings) $\cdots$ $[embed_1, embed_2, ..., embed_n]$ (business) where each $embed_i \in \mathbb{R}^{100}$. $\endgroup$ Commented Jan 7, 2023 at 15:58
  • 1
    $\begingroup$ But is there any meaningful ordering in the raw data? $\endgroup$ Commented Jan 7, 2023 at 16:42
  • $\begingroup$ I think yes due to the format of the raw data (Tree representation of text embedded). Unfortunately, the whole text is embedded into one vector of size 100. I was thinking of splitting the 100 into 4 vectors each of size 25 and then try to add positional encoding to the, but not sure if this is wise though? $\endgroup$ Commented Jan 8, 2023 at 0:32
  • 1
    $\begingroup$ No theoretically grounded, but you never know if it actually works until you try ¯_(ツ)_/¯ $\endgroup$ Commented Jan 8, 2023 at 8:01
  • $\begingroup$ Thank you. I was thinking that maybe the tree structure of the text before it was embedded into a 100-dimensional vector has some sort of order! $\endgroup$ Commented Jan 8, 2023 at 16:19

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.