Neural Network Rules

[gulp21]

@rob

Is it sufficient to just have the “dictionary.txt” and final “final_embeddings.txt” files that are in the various word2vec archives to train new “confusion_sets.txt” candidates?

Would the average developer not also need the whole set of setences that were used in creation of those files? That is - would the language specific wiki dump XML file and tatoeba sentences etc. used when making the dictionary/embeddings not also be required?

To be specific… Would the “training-corpus.txt” file not need to match? I guess another training corpus could be used, but then the dictionary may not contain all the words used in the corpus - is this an issue?

You do not need the original training-corpus.txt for determining the thresholds in confusion_sets.txt (in fact, in machine learning you never use the same corpus for training and tuning the parameters of a trained model). Out-of-vocabulary words (i.e. words which are not part of the dictionary) are no problem, since they are mapped to a special “UNKOWN” token.

[BebraiPuola]

Thanks for great work. It seems that commandline version with word2vec works faster than with n-gram, but server version for me is about 6 times slower.