Create a dictionary for French language and use it for spell checking

pecstef · May 17, 2016, 4:06pm

Now it works ! Many thanks !
Just another and last question: in order to end the process of creation of the french dictionary for the spell checker, I should follow the instructions of “developers” section in the wiki, shouldn’t I ?

dnaber · May 17, 2016, 7:16pm

If you already have a word list, you can do what’s described at “For Users” (Spell check - LanguageTool Wiki). The script mentioned in the “For Developers” section does almost the same, only that it first creates a list of words from the hunspell dictionary.

pecstef · May 18, 2016, 8:56am

Hello again,
I was maybe too much optimist!
I noticed that after creating the dictionary, its dimension is 1KB and my original dictionary.txt is 3.646 KB instead. Is this normal?

Then I tried to convert the just created dictionary.dict, in the related word list (in order to see if the content is the same as my word list) following the instruction of the wiki… But I can’t as you can see in the code below.

I copied the “french.info” file required, from org\languagetool\resource\fr… Maybe Do I need another .info file?

I noticed also that in org\languagetool\resource\fr there’s already a french dictionary.dict and after doing the same process, it returned a corrected word list.

Here’s my code in command line:

Here I created my dictionary using my word list:
> C:\Users\HP\Desktop\LT>java -cp languagetool.jar org.languagetool.tools.SpellDictionaryBuilder fr_FR -i C:/Users/HP/Desktop/LT/french.txt -info C:/Users/HP/Desktop/LT/french.info -o C:/Users/HP/Desktop/LT/output.dict
> Running Morfologik FSACompile.main with these options: [–exit, false, -i, C:\Users\HP\AppData\Local\Temp\SpellDictionaryBuilder781820826994572409.txt, -o, C:\Users\HP\Desktop\LT\output.dict, -f, CFSA2]
> Done. The binary dictionary has been written to C:\Users\HP\Desktop\LT\output.dict

Then I tried to re-convert the dictionary in a word list to see its content:

> C:\Users\HP\Desktop\LT>java -cp languagetool.jar org.languagetool.tools.DictionaryExporter -i C:/Users/HP/Desktop/LT/output.dict -info C:/Users/HP/Desktop/LT/french.info -o C:/Users/HP/Desktop/LT/test.dict
> Running Morfologik DictDecompile.main with these options: [--exit, false, -i, C:\Users\HP\Desktop\LT\output.dict, -o, C:\Users\HP\AppData\Local\Temp\DictionaryExporter_separator1455210922122033918.txt, --overwrite]
 > _> An unhandled exception occurred. Stack trace below.

_ > java.nio.file.NoSuchFileException: C:\Users\HP\Desktop\LT\output.info
_ > at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
_ > at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
_ > at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
_ > at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(Unknown Source)
_ > at java.nio.file.Files.newByteChannel(Unknown Source)
_ > at java.nio.file.Files.newByteChannel(Unknown Source)
_ > at java.nio.file.spi.FileSystemProvider.newInputStream(Unknown Source)
_ > at java.nio.file.Files.newInputStream(Unknown Source)
_ > at morfologik.stemming.Dictionary.read(Dictionary.java:64)
_ > at morfologik.tools.DictDecompile.call(DictDecompile.java:62)
_ > at morfologik.tools.DictDecompile.call(DictDecompile.java:20)
_ > at morfologik.tools.CliTool.main(CliTool.java:133)
_ > at morfologik.tools.DictDecompile.main(DictDecompile.java:132)
_ > at org.languagetool.tools.DictionaryExporter.build(DictionaryExporter.java:80)
_ > at org.languagetool.tools.DictionaryExporter.main(DictionaryExporter.java:59)
Done. The dictionary export has been written to C:/Users/HP/Desktop/LT/test.dict

pecstef · May 18, 2016, 9:19am

Update

I changed the encoding of my word list, I created a .info file and after executed the code, it created a file .dict of 205 KB.
I tried to open this file but the content is not readable.
How can I check if the .dict file has been well created?

arysin · May 18, 2016, 7:59pm

Theoretically the fastest way to check if your dictionary was created right is by using morfologik tools:

github.com

morfologik/morfologik-stemming/blob/master/morfologik-tools/src/main/package/examples/02-fsa-dump.txt

# This example dumps byte sequences from a finite
# state automaton (created in a previous example), 
# separating each sequence with a CR byte.

java -jar ../lib/${project.artifactId}-${project.version}.jar fsa_dump --input 01-fsa-build.fsa5 --output 02-fsa-dump.output

But for that you’ll need to clone and build that tool.

pecstef · May 24, 2016, 2:14pm

Since I got it, I would like to post here step by step how to create the dictionary and how to use it, for people who will need it again.

Hello this is my experience in creating a dictionary for spell checking with Language Tool ! Hope you enjoy it.

Part 1: How to create the dictionary

You need:

• A .txt file with the dictionary inside

• An .info file specifying the info on how to set LT output file (It is already present in LT directory).

• LanguageTool standalone version

• Java 8

At the end of this section, you will have:

• a .dict file i.e. the file with your dictionary in a readable form for LT

Install the LAST version of LT: Index of /snapshots/
Be sure to have your .txt in the right format (a) and encoding (b):
a. 1 word par line
b. UTF8 encoding
In the command line write:
a. java -cp languagetool.jar org.languagetool.tools.SpellDictionaryBuilder fr_FR -i path of the dictionary file -info path of the .info file -o path of the output file

where:

i. fr_FR is the code related to the language of the dictionary

ii. –i it’s the parameter of the input file (your .txt)

iii. –info it’s the parameter of the .info file related to the dictionary. You can create it following these instructions (Spell check - LanguageTool Wiki - “Configuring the dictionary” section) or use the .info already present – if present – in \org\languagetool\resource\yourlanguage

iv. –o it’s the parameter for specifing where you wish to save the .dict output file

Part 2: How to integrate the dictionary on LT for spell checking

You need:

• JDK 1.8 (http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)

• Maven (Maven – Download Apache Maven)

• IDE for Java (JetBrains, Eclipse, etc.)

• .info file + .dict file (see part1)

• GitHub LanguageTool project (GitHub - languagetool-org/languagetool: Style and Grammar Checker for 25+ Languages)

Set the JDK and Maven bin path (more info: Maven – Installing Apache Maven)
Copy the .info and .dict files created on part1 in \languagetool-master\languagetool-language-modules\YourLanguage\src\main\resources\org\languagetool\resource\YourLanguage\hunspell
Open with your IDE the java file called as the language of your dictionary (for ex. French.java) :

a. Change HunspellNoSuggestionRule in YourLanguage.java to MorfologikYourLanguageSpellerRule

 @Override
  public List<Rule> getRelevantRules(ResourceBundle messages) throws IOException {
    return Arrays.asList(
new CommaWhitespaceRule(messages),
new DoublePunctuationRule(messages),
new GenericUnpairedBracketsRule(messages,
Arrays.asList("[", "(", "{" /*"«", "‘"*/),
Arrays.asList("]", ")", "}"
/*"»", French dialog can contain multiple sentences. */
/*"’" used in "d’arm" and many other words */)),
new MorfologikYourLanguageSpellerRule(messages, this),
new UppercaseSentenceStartRule(messages, this),
new MultipleWhitespaceRule(messages, this),
new SentenceWhitespaceRule(messages),
// specific to French:
new CompoundRule(messages),
new QuestionWhitespaceRule(messages)
);
}

b. Create the new MorfologikYourLanguageSpellerRule.java in \languagetool-master\languagetool-language-modules\YourLanguage\src\main\java\org\languagetool\rules\YourLanguage :

/* LanguageTool, a natural language style checker
 * Copyright (C) 2012 Marcin Miłkowski (http://www.languagetool.org)
 *
 * This library is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public
 * License as published by the Free Software Foundation; either
 * version 2.1 of the License, or (at your option) any later version.
 *
 * This library is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 * Lesser General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public
 * License along with this library; if not, write to the Free Software
 * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301
 * USA
 */

package org.languagetool.rules.fr;

import java.io.IOException;
import java.util.ResourceBundle;

import org.languagetool.Language;
import org.languagetool.rules.spelling.morfologik.MorfologikSpellerRule;

public final class MorfologikYourLanguageSpellerRule extends MorfologikSpellerRule {

    public static final String RULE_ID = "MORFOLOGIK_RULE_CODEOFYOURLANGUAGE"; /* for ex. Fr_FR for French */

    private static final String RESOURCE_FILENAME = "PATH TO YOUR .DICT FILE";

	public MorfologikFrenchSpellerRule(ResourceBundle messages,
                                      Language language) throws IOException {
    super(messages, language);
  }

    @Override
    public String getFileName() {
        return RESOURCE_FILENAME;
    }

    @Override
    public String getId() {
        return RULE_ID;
    }
}

c. Go to \languagetool-master\ with your command line and write : Mvn package

d. See your results in \languagetool-master\languagetool-standalone\target\LanguageTool-3.4-SNAPSHOT\LanguageTool-3.4-SNAPSHOT.