Handling out-of-vocabulary word in word embedding

1 visualización (últimos 30 días)
Ismat Mohd Sulaiman
Ismat Mohd Sulaiman el 5 de Jul. de 2021
Comentada: Ismat Mohd Sulaiman el 16 de Ag. de 2021
I'm using FastText and my own word embedding on a set of documents. It is being used to detect abbreviations (Y/N) for each word token.
When testing, words that does not have vectors (out-of-vocabulary - OOV words), and discarded and not included in the performance measures (precision, recall, etc.) giving a false result. How do you handle this?
Would you replace all words with NaN values be included in the performance measure? Can the NaN values be replaced with a vector? How would you decide which vector?

Respuestas (1)

Prince Kumar
Prince Kumar el 16 de Ag. de 2021
From my understanding your want to handle OOV(out-of-vocabulary) words for your abbreviations detection task. For now MATLAB fastTextWordEmbedding does not handle OOV words.
There are many ways to do it, following are the two popular ones:
  1. Use context around the OOV word. You can use the word embedding of the previous and next word to your current OOV word.
  2. Use synonyms or similar word to get the word embedding for your OOV word.
  1 comentario
Ismat Mohd Sulaiman
Ismat Mohd Sulaiman el 16 de Ag. de 2021
Thank you! I have used the first approach. Excited to try or add on the 2nd approach.

Iniciar sesión para comentar.

Categorías

Más información sobre Get Started with MATLAB en Help Center y File Exchange.

Productos


Versión

R2020b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by