and converted it to the .dict format for the LanguageTool using:
the standalone “languagetool.jar”;
the “english.info” from the resources area in the unarchived languagetool:
languagetool\languagetool-language-modules\en\src\main\resources\org\languagetool\resource\en
The command line (Linux):
java -cp langaugetool.jar:morfologik-tools-1.8.1-standalone.jar:morfologik-tools-1.5.2-standalone.jar org.languagetool.dev.POSDictionaryBuilder …/oec_dict4languagetool.txt english.info
which ran fine and created a temporary file that I renamed as “mydic.dict”.
The same for the synthetizer dictionary, using org.languagetool.dev.SynthDictionaryBuilder.
The tutorial for building a customized dictionary (link above) does not specify where to save the .dict file in the languagetool system in order to use it.
Thank you, I’ve overwritten the english.dict and english_synth.dict files with the custom dictionaries. However, I don’t know how to use it now in an Eclipse project. I’ve tried to create a jar file of the languagetool folder but it contains everything inclusive of languagetool-core folder, languagetool-language-models folder etc… The command “import languagetool.LanguageTool;” in a new Java class file that tries to use the customized lemmatizer doesn’t work. What to do? Can you help again?
Many thanks!
With “import languagetool.LanguageTool;” you’re trying to import a class named “LanguageTool” that’s in a package named “languagetool”. I guess that’s not what you want? You need to point your Java classpath to all the JARs that come with LanguageTool (the one you modified and the other ones). I don’t use Eclipse, so sorry I cannot be more specific.
and replaced the english.dict and english_synth.dict inside the folder: /languagetool/languagetool-language-modules/en/src/main/resources/org/languagetool/resource/en
I still cannot use the new dictionary. I tried using the maven command line to build it but I get this error:
C:\Users<my_user>.m2\repository\de\danielnaber\jwordsplitter\4.1\jwordsplitter-4.1.jar <my_path>\languagetool\languagetool-language-modules\de-DE-x-simple-language\target\classes <my_path>\languagetool\languagetool-language-modules\pl\target\classes <my_path>\languagetool\languagetool-language-modules\ca\target\classes <my_path>\languagetool\languagetool-language-modules\it\target\classes <my_path>\languagetool\languagetool-language-modules\br\target\classes <my_path>\languagetool\languagetool-language-modules\nl\target\classes <my_path>\languagetool\languagetool-language-modules\pt\target\classes <my_path>\languagetool\languagetool-language-modules\ru\target\classes <my_path>\languagetool\languagetool-language-modules\ast\target\classes <my_path>\languagetool\languagetool-language-modules\be\target\classes <my_path>\languagetool\languagetool-language-modules\zh\target\classes C:\Users<my_user>.m2\repository\com\googlecode\ictclas4j\ictclas4j\1.0.1\ictclas4j-1.0.1.jar C:\Users<my_user>.m2\repository\com\google\code\cjftransform\1.0.1\cjftransform-1.0.1.jar <my_path>\languagetool\languagetool-language-modules\da\target\classes <my_path>\languagetool\languagetool-language-modules\eo\target\classes <my_path>\languagetool\languagetool-language-modules\gl\target\classes <my_path>\languagetool\languagetool-language-modules\el\target\classes <my_path>\languagetool\languagetool-language-modules\is\target\classes <my_path>\languagetool\languagetool-language-modules\ja\target\classes C:\Users<my_user>.m2\repository\com\google\code\lucene-gosen-ipadic\1.2.1\lucene-gosen-ipadic-1.2.1.jar <my_path>\languagetool\languagetool-language-modules\km\target\classes <my_path>\languagetool\languagetool-language-modules\lt\target\classes <my_path>\languagetool\languagetool-language-modules\ml\target\classes <my_path>\languagetool\languagetool-language-modules\ro\target\classes <my_path>\languagetool\languagetool-language-modules\sk\target\classes <my_path>\languagetool\languagetool-language-modules\sl\target\classes <my_path>\languagetool\languagetool-language-modules\es\target\classes <my_path>\languagetool\languagetool-language-modules\sv\target\classes <my_path>\languagetool\languagetool-language-modules\ta\target\classes <my_path>\languagetool\languagetool-language-modules\tl\target\classes <my_path>\languagetool\languagetool-language-modules\uk\target\classes C:\Users<my_user>.m2\repository\org\slf4j\slf4j-nop\1.7.9\slf4j-nop-1.7.9.jar C:\Users<my_user>.m2\repository\org\slf4j\slf4j-api\1.7.9\slf4j-api-1.7.9.jar <my_path>\languagetool\languagetool-core\target\test-classes C:\Users<my_user>.m2\repository\junit\junit\4.12\junit-4.12.jar C:\Users<my_user>.m2\repository\org\hamcrest\hamcrest-core\1.3\hamcrest-core-1.3.jar C:\Users<my_user>.m2\repository\org\apache\maven\surefire\surefire-junit4\2.18.1\surefire-junit4-2.18.1.jar
[DEBUG] boot(compact) classpath: surefire-booter-2.18.1.jar surefire-api-2.18.1.jar test-classes classes classes commons-lang-2.6.jar jna-4.1.0.jar morfologik-fsa-1.10.0.jar morfologik-speller-1.10.0.jar morfologik-stemming-1.10.0.jar hppc-0.6.1.jar segment-1.4.2.jar commons-logging-1.1.1.jar language-detector-0.5.jar jsonic-1.2.11.jar guava-16.0.1.jar annotations-12.0.jar lucene-core-5.2.1.jar lucene-backward-codecs-5.2.1.jar classes classes opennlp-tools-1.6.0.jar opennlp-tokenize-models-1.5.jar opennlp-postag-models-1.5.jar opennlp-chunk-models-1.5.jar classes classes hunspell-native-libs-2.9.jar classes openregex-1.1.1.jar jwordsplitter-4.1.jar classes classes classes classes classes classes classes classes classes classes classes ictclas4j-1.0.1.jar cjftransform-1.0.1.jar classes classes classes classes classes classes lucene-gosen-ipadic-1.2.1.jar classes classes classes classes classes classes classes classes classes classes classes slf4j-nop-1.7.9.jar slf4j-api-1.7.9.jar test-classes junit-4.12.jar hamcrest-core-1.3.jar surefire-junit4-2.18.1.jar
Forking command line: cmd.exe /X /C ““C:\Program Files\Java\jdk1.8.0_60\jre\bin\java” -Xms256m -Xmx700m -jar <my_path>\languagetool\languagetool-commandline\target\surefire\surefirebooter497822894376538476.jar <my_path>\languagetool\languagetool-commandline\target\surefire\surefire727518009555671302tmp <my_path>\languagetool\languagetool-commandline\target\surefire\surefire_344409416620683103884tmp”
Running org.languagetool.commandline.MainTest
Tests run: 41, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 14.354 sec <<< FAILURE! - in org.languagetool.commandline.MainTest
testPolishApiStdInDefaultOff(org.languagetool.commandline.MainTest) Time elapsed: 0.125 sec <<< FAILURE!
java.lang.AssertionError:
Expected: is <1>
but: was <2>
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at org.languagetool.commandline.MainTest.testPolishApiStdInDefaultOff(MainTest.java:333)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:252)
at junit.framework.TestSuite.run(TestSuite.java:247)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Running org.languagetool.commandline.CommandLineToolsTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in org.languagetool.commandline.CommandLineToolsTest
Running org.languagetool.commandline.CommandLineParserTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in org.languagetool.commandline.CommandLineParserTest
Results :
Failed tests:
MainTest.testPolishApiStdInDefaultOff:333
Expected: is <1>
but: was <2>
Tests run: 45, Failures: 1, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] languagetool-parent … SUCCESS [ 0.342 s]
[INFO] LanguageTool Style and Grammar Checker Core … SUCCESS [ 4.369 s]
[INFO] English module for LanguageTool … SUCCESS [ 46.508 s]
[INFO] Persian module for LanguageTool … SUCCESS [ 3.381 s]
[INFO] French module for LanguageTool … SUCCESS [ 13.970 s]
[INFO] German module for LanguageTool … SUCCESS [ 44.056 s]
[INFO] Polish module for LanguageTool … SUCCESS [ 29.426 s]
[INFO] Catalan module for LanguageTool … SUCCESS [ 48.460 s]
[INFO] Italian module for LanguageTool … SUCCESS [ 1.768 s]
[INFO] Breton module for LanguageTool … SUCCESS [ 3.968 s]
[INFO] Dutch module for LanguageTool … SUCCESS [ 4.461 s]
[INFO] Portuguese module for LanguageTool … SUCCESS [ 2.062 s]
[INFO] Russian module for LanguageTool … SUCCESS [ 5.090 s]
[INFO] Asturian module for LanguageTool … SUCCESS [ 1.158 s]
[INFO] Belarusian module for LanguageTool … SUCCESS [ 0.722 s]
[INFO] Chinese module for LanguageTool … SUCCESS [ 5.436 s]
[INFO] Danish module for LanguageTool … SUCCESS [ 1.361 s]
[INFO] Esperanto module for LanguageTool … SUCCESS [ 4.153 s]
[INFO] Galician module for LanguageTool … SUCCESS [ 1.501 s]
[INFO] Greek module for LanguageTool … SUCCESS [ 0.798 s]
[INFO] Icelandic module for LanguageTool … SUCCESS [ 0.783 s]
[INFO] Japanese module for LanguageTool … SUCCESS [ 1.797 s]
[INFO] Khmer module for LanguageTool … SUCCESS [ 1.158 s]
[INFO] Lithuanian module for LanguageTool … SUCCESS [ 0.970 s]
[INFO] Malayalam module for LanguageTool … SUCCESS [ 1.095 s]
[INFO] Romanian module for LanguageTool … SUCCESS [ 3.232 s]
[INFO] Slovak module for LanguageTool … SUCCESS [ 2.749 s]
[INFO] Slovenian module for LanguageTool … SUCCESS [ 1.079 s]
[INFO] Spanish module for LanguageTool … SUCCESS [ 2.096 s]
[INFO] Swedish module for LanguageTool … SUCCESS [ 1.298 s]
[INFO] Tamil module for LanguageTool … SUCCESS [ 1.239 s]
[INFO] Tagalog module for LanguageTool … SUCCESS [ 1.126 s]
[INFO] Ukrainian module for LanguageTool … SUCCESS [ 6.181 s]
[INFO] Simple German module for LanguageTool … SUCCESS [ 1.606 s]
[INFO] All language modules for LanguageTool … SUCCESS [ 0.031 s]
[INFO] LanguageTool common GUI classes … SUCCESS [ 0.486 s] [INFO] LanguageTool command-line version … FAILURE [ 15.840 s]
[INFO] LanguageTool embedded HTTP server … SKIPPED
[INFO] LanguageTool stand-alone GUI … SKIPPED
[INFO] LanguageTool LibreOffice/OpenOffice extension … SKIPPED
[INFO] Hunspell native libs for LanguageTool … SKIPPED
[INFO] LanguageTool Wikipedia tools … SKIPPED
[INFO] LanguageTool development tools … SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:26 min
[INFO] Finished at: 2015-10-06T15:37:44+01:00
[INFO] Final Memory: 19M/321M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test (default-test) on project languagetool-commandline: There are test failures.
[ERROR]
Any idea on the error above? And How to recompile the project? I couldn’t find any reference on how to rebuild the languagetool and use a custom dictionary after creating the new dictionary.
Could you build LT without your changes? Which dictionary have you modified, I assume the Polish one? The message says that 2 error where found where 1 is expected. You could build your changes by skipping the tests for now: “mvn package -DskipTests”
I still get an error when building the LT without any change (unzipped the archive from github again):
ipse-workspace/LT/languagetool
$ mvn package -DskipTests
[INFO] Scanning for projects…
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] languagetool-parent
[INFO] LanguageTool Style and Grammar Checker Core
[INFO] English module for LanguageTool
[INFO] Persian module for LanguageTool
[INFO] French module for LanguageTool
[INFO] German module for LanguageTool
[INFO] Polish module for LanguageTool
[INFO] Catalan module for LanguageTool
[INFO] Italian module for LanguageTool
[INFO] Breton module for LanguageTool
[INFO] Dutch module for LanguageTool
[INFO] Portuguese module for LanguageTool
[INFO] Russian module for LanguageTool
[INFO] Asturian module for LanguageTool
[INFO] Belarusian module for LanguageTool
[INFO] Chinese module for LanguageTool
[INFO] Danish module for LanguageTool
[INFO] Esperanto module for LanguageTool
[INFO] Galician module for LanguageTool
[INFO] Greek module for LanguageTool
[INFO] Icelandic module for LanguageTool
[INFO] Japanese module for LanguageTool
[INFO] Khmer module for LanguageTool
[INFO] Lithuanian module for LanguageTool
[INFO] Malayalam module for LanguageTool
[INFO] Romanian module for LanguageTool
[INFO] Slovak module for LanguageTool
[INFO] Slovenian module for LanguageTool
[INFO] Spanish module for LanguageTool
[INFO] Swedish module for LanguageTool
[INFO] Tamil module for LanguageTool
[INFO] Tagalog module for LanguageTool
[INFO] Ukrainian module for LanguageTool
[INFO] Simple German module for LanguageTool
[INFO] All language modules for LanguageTool
[INFO] LanguageTool common GUI classes
[INFO] LanguageTool command-line version
[INFO] LanguageTool embedded HTTP server
[INFO] LanguageTool stand-alone GUI
[INFO] LanguageTool LibreOffice/OpenOffice extension
[INFO] Hunspell native libs for LanguageTool
[INFO] LanguageTool Wikipedia tools
[INFO] LanguageTool development tools
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building languagetool-parent 3.2-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] — maven-enforcer-plugin:1.0:enforce (enforce-maven) @ languagetool-parent —
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building LanguageTool Style and Grammar Checker Core 3.2-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] — maven-enforcer-plugin:1.4.1:enforce (enforce-maven) @ languagetool-core —
[INFO]
[INFO] — maven-resources-plugin:2.6:resources (default-resources) @ languagetool-core —
[INFO] Using ‘UTF-8’ encoding to copy filtered resources.
[INFO] Copying 15 resources
[INFO]
[INFO] — maven-compiler-plugin:3.2:compile (default-compile) @ languagetool-core —
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 150 source files to <my_path>\languagetool\languagetool-core\target\classes
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] languagetool-parent … SUCCESS [ 0.372 s] [INFO] LanguageTool Style and Grammar Checker Core … FAILURE [ 1.118 s]
[INFO] English module for LanguageTool … SKIPPED
[INFO] Persian module for LanguageTool … SKIPPED
[INFO] French module for LanguageTool … SKIPPED
[INFO] German module for LanguageTool … SKIPPED
[INFO] Polish module for LanguageTool … SKIPPED
[INFO] Catalan module for LanguageTool … SKIPPED
[INFO] Italian module for LanguageTool … SKIPPED
[INFO] Breton module for LanguageTool … SKIPPED
[INFO] Dutch module for LanguageTool … SKIPPED
[INFO] Portuguese module for LanguageTool … SKIPPED
[INFO] Russian module for LanguageTool … SKIPPED
[INFO] Asturian module for LanguageTool … SKIPPED
[INFO] Belarusian module for LanguageTool … SKIPPED
[INFO] Chinese module for LanguageTool … SKIPPED
[INFO] Danish module for LanguageTool … SKIPPED
[INFO] Esperanto module for LanguageTool … SKIPPED
[INFO] Galician module for LanguageTool … SKIPPED
[INFO] Greek module for LanguageTool … SKIPPED
[INFO] Icelandic module for LanguageTool … SKIPPED
[INFO] Japanese module for LanguageTool … SKIPPED
[INFO] Khmer module for LanguageTool … SKIPPED
[INFO] Lithuanian module for LanguageTool … SKIPPED
[INFO] Malayalam module for LanguageTool … SKIPPED
[INFO] Romanian module for LanguageTool … SKIPPED
[INFO] Slovak module for LanguageTool … SKIPPED
[INFO] Slovenian module for LanguageTool … SKIPPED
[INFO] Spanish module for LanguageTool … SKIPPED
[INFO] Swedish module for LanguageTool … SKIPPED
[INFO] Tamil module for LanguageTool … SKIPPED
[INFO] Tagalog module for LanguageTool … SKIPPED
[INFO] Ukrainian module for LanguageTool … SKIPPED
[INFO] Simple German module for LanguageTool … SKIPPED
[INFO] All language modules for LanguageTool … SKIPPED
[INFO] LanguageTool common GUI classes … SKIPPED
[INFO] LanguageTool command-line version … SKIPPED
[INFO] LanguageTool embedded HTTP server … SKIPPED
[INFO] LanguageTool stand-alone GUI … SKIPPED
[INFO] LanguageTool LibreOffice/OpenOffice extension … SKIPPED
[INFO] Hunspell native libs for LanguageTool … SKIPPED
[INFO] LanguageTool Wikipedia tools … SKIPPED
[INFO] LanguageTool development tools … SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.081 s
[INFO] Finished at: 2015-10-06T16:38:40+01:00
[INFO] Final Memory: 17M/310M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project languagetool-core: Fatal error compiling: invalid target release: 1.8 → [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] MojoExecutionException - Apache Maven - Apache Software Foundation
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :languagetool-core
What else to do?
My aim: I want to use a different English dictionary as part of an nlp pipeline using the LT lemmatizer.
However, how to call the Lemmatizer? I cannot find any class with a relevant name in these jars. I tried creating first an instance of the LT:
JLanguageTool lt = new JLanguageTool(new English());
But how to replace the line below (that used the previous LT directly from the Maven repository, through the DKPro framework) to point to the new lemmatizer?
lemmatize = createEngineDescription( LanguageToolLemmatizer.class );
To use the lemmatizer, create a language (e.g. “new English()”) and then get the lemmatizer with the getSentenceTokenizer() or getWordTokenizer() methods. To use you own lemmatizer, overwrite one or both of these methods in the language class.