[BUG] — Spell checking is disabled for every language without n-gram data when languageModel is set (regression since 6.6, #11226)

Development
1

Summary

Since 6.6, an HTTP server started with languageModel set loses the spell checker for every language that has no n-gram data in that directory. MORFOLOGIK_RULE_EN_* and GERMAN_SPELLER_RULE no longer return any matches, with nothing in the log beyond the usual WARN: ngram index dir … not found. In 6.4 the same configuration keeps the spell checkers working.

This hits a common setup: a server with n-gram data for one language only. With only ngrams/es installed, Spanish spelling still works, but English spelling is gone completely.

Steps to reproduce

An empty n-gram directory is enough; no n-gram download is needed.

mkdir /tmp/empty-ngrams
echo "languageModel=/tmp/empty-ngrams" > /tmp/lt.properties

From the unzipped LanguageTool-6.6 folder:

java -cp "languagetool-server.jar:libs/*" org.languagetool.server.HTTPServer --port 8081 --config /tmp/lt.properties

curl -s --data-urlencode "text=We will meet tomorow at the libary." -d language=en-US http://localhost:8081/v2/check

Expected: two MORFOLOGIK_RULE_EN_US matches (“tomorow”, “libary”), as in 6.4, or in 6.6 without languageModel.

Actual: no matches.

Results

The same sentence, plus “Wir treffen uns morgn in der Bibliotek.” for German. Each cell is the number of spelling matches.

Version languageModel en-US en-GB de-DE
6.4 not set 2 2 2
6.4 empty directory 2 2 2
6.6 not set 2 2 2
6.6 empty directory 0 0 0
6.8 not set 2 2 2
6.8 empty directory 0 0 0

6.8 was run from the languagetool-server 6.8 and language-all 6.8 artifacts on Maven Central. The command-line tool of 6.6 flags both English words, so the dictionaries themselves are fine. Tested on macOS with OpenJDK 21.

Cause

#11226 added a withLanguageModel flag. When it is true, the JLanguageTool constructor no longer adds the rules that can optionally use a language model, without a model (JLanguageTool.java#L360-L361):

if (!withLanguageModel) {
  updateOptionalLanguageModelRules(null); // start out with rules without language model
}

The server passes true whenever languageModel is configured, then calls activateLanguageModelRules (PipelinePool.java#L140-L141). That method only adds those rules when the language’s model actually loads (JLanguageTool.java#L614-L620). For a language with no <languageModel>/<code> directory, initLanguageModel returns null (LanguageWithModel.java#L36-L43), so updateOptionalLanguageModelRules is never called.

The English and German spell checkers are among those optional rules. For example, BritishEnglish creates its speller in getRelevantLanguageModelCapableRules (BritishEnglish.java#L82-L84). So these languages end up with no spell checker at all.

Possible fix

Add the optional rules even when no model loads for the language, as the constructor did before #11226:

public void activateLanguageModelRules(File indexDir) throws IOException {
  LanguageModel languageModel = language.getLanguageModel(indexDir);
  if (languageModel != null) {
    ResourceBundle messages = getMessageBundle(language);
    List<Rule> rules = language.getRelevantLanguageModelRules(messages, languageModel, userConfig);
    userRules.addAll(rules);
  }
  // Rules that only optionally use a model (such as the spellers) must exist either way.
  updateOptionalLanguageModelRules(languageModel);
}

Workaround

Either leave languageModel unset, or install n-gram data for every language the server checks.