I found a very interesting thread at stackoverflow explaining how hunspell works. Since Languagetool is using hunspell, the explanation there applies to the Languagetool spelling checker:
This is a good example of why the Unicode standard is important. The U+0027 apostrophe character is punctuation, not intended to be part of a word. Use the word-forming code point U+02BC instead.
So instead of using the disambiguation rule I can simple add the word using the word-forming code point ‘MODIFIER LETTER APOSTROPHE’:
Old: Bahá’í
New: Baháʼí
You need a hex editor to see that the two apostrophes are actually different!
So now I can add the new form to spelling.txt and the spell checker can handle it, either alone or as a part of compound words.