Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: error (nike.apache.org: local policy)
From: Karl Wettin <karl.wettin@kodapan.se>
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
Date: Wed, 22 May 2013 14:37:46 +0200
Subject: =?iso-8859-1?Q?Bl=E5b=E6rsyltet=F8y_v=2Es=2E_R=E4ksm=F6rg=E5s?=
To: java-user@lucene.apache.org
Message-Id: <DAF6466E-C34D-47A5-989F-B383D0614C5F@kodapan.se>
Mime-Version: 1.0 (Apple Message framework v1283)

This is a question (or perhaps a line of thought) regarding the mutually =
intelligible Scandinavian languages Danish, Norwegian and Swedish.


The Swedish letters =E5=E4=F6 is in fact the same letters as the =
Danish/Norwegian =E5=E6=F8. A Norwegian writing about the Swedish city =
of G=F6teborg write G=F8teborg and a Swedish person writing about =
Svolv=E6r will write Svolv=E4r. This is easy to fix, I can just index =
synonyms where =E4=F6 is replaced by =E6=F8 and vice verse.

More problematic, at least in my head, is ASCII-folding.

When a Swedish person is lacking umlauted characters on the keyboard =
they consistently type a, a, o instead of =E5, =E4, =F6. Foreigners also =
tend to use a, a, o.=20

In Norway people tend to type aa, ae and oe instead of =E5, =E6 and =F8. =
Some use a, a, o. I've also seen oo, ao, etc. And permutations. Not sure =
about Denmark but the pattern is probably the same. I have no clue to =
what letters foreigners might be replacing them with.


There's a lot of mismatch here. For a start ASCIIFoldingFilter translate =
'=E4' to 'a' and '=E6' as 'ae'. The rest is not aligned with what people =
actually type, such as '=F8' to 'o' rather than the more common 'oe'.


I'm considering:

* Forking ASCIIFoldingFilter with a bunch of strategies and index =
permutations of synonyms.
or
* Use a filter after ASCIIFoldingFilter that discriminate all use of ae, =
oe, oo, and other combination of double vowels, just keeping the first =
one.


Anyone else that thought about this?


			karl=

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org