Mailing-List: contact general-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of hossman_lucene@fucit.org
 designates 208.69.42.181 as permitted sender)
Date: Mon, 22 Nov 2010 11:58:27 -0800 (PST)
From: Chris Hostetter <hossman_lucene@fucit.org>
To: general@lucene.apache.org
Subject: 
 =?ISO-8859-15?Q?Re=3A_special_characters_=22=F8=22_indexing=2Fsearching?=
In-Reply-To: <AANLkTikS+EHBYUTOspi0Au4h2iHUZq8+X4TsvGn3Bo=e@mail.gmail.com>
Message-ID: <alpine.DEB.1.10.1011221154310.19563@radix.cryptio.net>
References: <1289829779397-1904492.post@n3.nabble.com>
 <1289920557074-1911347.post@n3.nabble.com>
 <alpine.DEB.1.10.1011191522270.8813@radix.cryptio.net>
 <AANLkTikQZCsJv0OWDO-fR1k6chzwbOkdACOBaY30QawD@mail.gmail.com>
 <alpine.DEB.1.10.1011191535030.8813@radix.cryptio.net>
 <AANLkTikrE1mQZ9pDoaUhq9G_NdU-cGAhwp=ABVR5HBNq@mail.gmail.com>
 <AANLkTikS+EHBYUTOspi0Au4h2iHUZq8+X4TsvGn3Bo=e@mail.gmail.com>
User-Agent: Alpine 1.10 (DEB 962 2008-03-14)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII


: I disagree with Hoss on this issue, removing diacritics in a filter is
: not going to "mess up highlighting". The offsets are set by the
: tokenizer. So its no different than stemming or any other process.

thanks for correcting me dude ... i'm not sure what i wsa thinkg of, but 
for some reason i thought there was an issue with the highlighter and 
token filters that changed the lengths of tokens (including stemming).

: The *only* situation where you should use a CharFilter, is when you
: must change this stuff before the tokenizer.

Can you elaborate on that, because it's definitely something that i'm 
getting more and more confused by, so i'm sure other people are confused 
as well.

what is an example of a situation where you "must" change stuff before the 
tokenizer?  the HTML Stripper is the one example i understand, but the 
purpose of hte mapping char filter no longer make sense to me in light of 
this thread.


-Hoss