Word2Vec confusion pairs

[Ferch42]

Hello everyone

I would like to help building the Word2Vec models for confusion pairs. For that I would have to find a large córpus with examples right?. How would that córpus be like? Just sentences with the confusion pairs? And how big does it need to be?

[Ruud_Baars]

If you would do so for Dutch, I can give a corpus.

[tiagosantos (Tiago F. Santos)]

Hi @Ferch42,

Maybe you have already seen this:

Neural Network Rules

Hi, as part of my project work at university, I looked into detecting confused words (e. g. to/too, than/then) using the word2vec model by Mikolov et al and neural networks (which get 5-grams as input). I’ve integrated some neural network word confusion rules into LanguageTool (source code) and a small online demo for German and English is available here. (The forms are sometimes buggy and in case my LanguageTool server crashes I have to restart it by hand.) [image] So in which way are the n…

Gulp21 has used a public corpus from magazines (the link is in the thread), that seems perfect for this task.
I would pass you my stuff but I am away from home and I am not using my development computer, so it will take a few months to pass to you the ‘starter’s kit’.

If you decide to do this for Portuguese I’ll be happy.

If possible use a corpus with only one language variant (just Brazilian Portuguese or just European Portuguese), even if not making a variant specific model, or you will have sub-optimal results in some circumstances.

I would argue against using Wikipedia as a corpus, since text have many typos and they mix both variants, plus the spelling agreement varieties, sometimes within the same sentence, possibly creating biases and errors on the models.

When training the model, the more the merrier, although it will be an extra burden processing more data.

Regarding this topic, @gulp21 is the expert and he will be the best person to give advice, so I recommend posting further enquiries on that thread.

[gulp21]

@Ferch42

For that I would have to find a large córpus with examples right?. How would that córpus be like? Just sentences with the confusion pairs? And how big does it need to be?

It should be a large corpus which contains almost no errors. In my experiments, I used corpora with more than 1 million sentences. It is ok if the corpus contains some errors, but not too many; so newspaper articles are a good candidate, but forum posts are not. The corpus file is a plain text file containing full sentences. Do not limit the corpus to a specific confusion pair, but try to cover a wide range of styles (e.g. newspaper articles, books, etc.), although it can be hard/impossible to get free corpora; I’ve only used newspaper articles and tatoeba and got good results (see the thread linked in the previous post).

By the way, it is probably a good idea to document (in a readme or maybe the Wiki) which sources has been used for a corpus. This can come in handy if someone wants to do further experiments with machine learning.

[Ferch42]

OK! Thanks. I will take a look into it.