lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject RE: Indexing puncutation
Date Wed, 29 Jun 2005 18:48:19 GMT


keep in mind, you can "store" the raw field for display purposes and
"index" many different token sequences that represent the same orriginal
data parsed in several ways -- all using the same field name.


: Date: Wed, 29 Jun 2005 13:33:42 -0400
: From: "Aigner, Thomas" <TAigner@WescoDist.com>
: Reply-To: java-user@lucene.apache.org
: To: java-user@lucene.apache.org
: Subject: RE: Indexing puncutation
:
: Thanks for the advice.  I have replaced punctuation before the index is
: built and then queried on the same lack of punctuation.  I had to create
: a separate index for this as well so I have the original information,
: but I think I will take your advice and build a custom token to filter
: out the punctuation but keep the contents the original.
:
: Tom
:
: -----Original Message-----
: From: Ken Krugler [mailto:kkrugler_lists@transpac.com]
: Sent: Wednesday, June 29, 2005 10:39 AM
: To: java-user@lucene.apache.org
: Subject: Re: Indexing puncutation
:
: >I do a vaguely similar thing;  I have to strip accents from
: >characters such as e-acute out of both my input data and my incoming
: >search queries to put them into a standard form.  I do this with a
: >custom TokenFilter subclass.  I have an analyzer that includes this
: >filter along with some of the standard ones (LowercaseFilter, etc).
: >I run the same analyzer on indexing and searching, which has been
: >discussed in other posts.
:
: For a hard-core approach to this problem, you could try converting
: all text to Unicode first, then use the ICU package to create a level
: 0 "sort key" (the C API is col_getSortKey). This will be a string
: suitable for comparison to determine weak equality, but you can also
: just index it as a regular token.
:
: There are some subtle issues w/locale-specific behavior of the sort
: key generation step, where you could guess at the right locale to use
: for the conversion, but in general that shouldn't matter.
:
: Two other issues are code/data size (ICU can be big) and the
: performance hit while indexing documents.
:
: -- Ken
:
:
:
: >Aigner, Thomas wrote:
: >
: >>Hello all,
: >>
: >>	I am VERY new to Lucene and we are trying out Lucene to see if
: >>it will accomplish the vast majority of our search functions.
: >>
: >>	I have a question about a good way to index some of our product
: >>description codes.  We have description codes like 21-MA-GAB and other
: >>punctuation.  Our users need to be able to search for "21 MA GAB"
: >>or "21-MA_GAB" or "21MAGAB".  Is the best way to accomplish this by
: >>creating synonyms for the 3 different ways when punctuation is in
: parts
: >>to search for? I know I can stop punctuation in the index but what
: about
: >>grouping the information together or with spaces?
: >>
: >>Thanks all in advance,
: >>Tom
:
:
: --
: Ken Krugler
: TransPac Software, Inc.
: <http://www.transpac.com>
: +1 530-470-9200
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: For additional commands, e-mail: java-user-help@lucene.apache.org
:
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: For additional commands, e-mail: java-user-help@lucene.apache.org
:



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message