lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Penney" <karl.pen...@eastech.ca>
Subject Re: Disabling modifiers?
Date Tue, 16 Dec 2003 13:58:28 GMT
One of the token patterns defined by the StandardTokenizer.jj is this:
<NUM: (<ALPHANUM> <P> <HAS_DIGIT>

| <HAS_DIGIT> <P> <ALPHANUM>

| <ALPHANUM> (<P> <HAS_DIGIT> <P> <ALPHANUM>)+

| <HAS_DIGIT> (<P> <ALPHANUM> <P> <HAS_DIGIT>)+

| <ALPHANUM> <P> <HAS_DIGIT> (<P> <ALPHANUM> <P> <HAS_DIGIT>)+

| <HAS_DIGIT> <P> <ALPHANUM> (<P> <HAS_DIGIT> <P> <ALPHANUM>)+

)

So basically if you have some sequences of characters separated by a "-"
character, sequences that contain a digit will be combined with sequences
which are adjacent to it to form a single token.  That explains why the WS
and YYMM sequences got separated out.  You can alter this behavior this with
some simple changes to StandardTokenizer.jj.

----- Original Message -----
From: "Iain Young" <Iain.Young@microfocus.com>
To: "'Lucene Users List'" <lucene-user@jakarta.apache.org>
Sent: Tuesday, December 16, 2003 7:46 AM
Subject: RE: Disabling modifiers?


> I think it is a problem with the indexing. I've found another example...
>
> WS-CA-PP00-PROCESS-YYMM
>
> I've looked at the index, and it has been tokenized into 3 words...
>
> WS
> CA-PP00-PROCESS
> YYMM
>
> Looks as though I might have to use a custom tokenizer as well as an
> analyzer then, but any ideas as to why the standard tokenizer would have
> split the variable up like this (i.e. why didn't it split the middle bit,
> only the word off either end)? The only thing I can think of is that there
> are several other variables in the source beginning with WS- or ending
with
> -YYMM, so could the tokenizer have seen this and be doing something clever
> with them?
>
> Thanks,
> Iain
>
> *****************************************
> *  Micro Focus Developer Forum 2004     *
> *  3 days that will make a difference   *
> *  www.microfocus.com/devforum          *
> *****************************************
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message