Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: domain of markrmiller@gmail.com designates
 66.249.92.170 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=Pe3UJ7vNONqWzXLAPmToP32R7IEQTgnMdoVrujCbJ+lWgYR6ccO0RsSjoGXEZBkS1rhqEEW0RIZGYsbdYxT2i39aF73lB6Rh7YRJxntj74X5yal4GaVpqQrwKFu1wp7EJO9TICAXuDr5UQ48vMpCjb+zCrd5jm5i6NFchLOwgRw=
Message-ID: <ab0709bf0607211309k1e3a8e24u85896582dfcf08ef@mail.gmail.com>
Date: Fri, 21 Jul 2006 16:09:28 -0400
From: "Mark Miller" <markrmiller@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: StandardAnalyzer question
In-Reply-To: <51540A3DDD507D40B6D47030C8C0C19101A3A1F1@soumaiexcp01.iss.net>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_9_20444770.1153512568127"
References: <51540A3DDD507D40B6D47030C8C0C19101A3A1F1@soumaiexcp01.iss.net>

------=_Part_9_20444770.1153512568127
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

| < #LETTER:                      // unicode letters
      [
       "\u0041"-"\u005a",
       "\u0061"-"\u007a",
       "\u00c0"-"\u00d6",
       "\u00d8"-"\u00f6",
       "\u00f8"-"\u00ff",
       "\u0100"-"\u1fff"
      ]

becomes

| < #LETTER:                      // unicode letters
      [
       "\u0041"-"\u005a",
       "\u0061"-"\u007a",
       "\u00c0"-"\u00d6",
       "\u00d8"-"\u00f6",
       "\u00f8"-"\u00ff",
       "\u0100"-"\u1fff",
       "\u002d"
      ]

On 7/21/06, Ngo, Anh (ISS Southfield) <ango@iss.net> wrote:
>
>
> Hello Mark,
>
>
> Please show me how to add "-" to #LETTER definition
>
>
> Thanks,
>
>
> Anh Ngo
>
> -----Original Message-----
> From: Mark Miller [mailto:markrmiller@gmail.com]
> Sent: Friday, July 21, 2006 3:51 PM
> To: java-user@lucene.apache.org
> Subject: Re: StandardAnalyzer question
>
> I do not beleive so. If you look above you will see that #P is only used
> when looking for a num: a host ip, a phone number, etc. You will be
> removing
> that ability to recognize a "_" while rooting those tokens out. It will
> still be parsed when tokenizing an EMAIL as well. I dont think this is
> the
> behavior you want.
>
> - Mark
>
> On 7/21/06, Ngo, Anh (ISS Southfield) <ango@iss.net> wrote:
> >
> >
> > What is #LETTER definition in SnardarTokernize.jj?
> >
> >
> > I saw:
> >
> > | <#P: ("_"|"-"|"/"|"."|",") >
> > | <#HAS_DIGIT:                                    // at least one
> digit
> >     (<LETTER>|<DIGIT>)*
> >     <DIGIT>
> >     (<LETTER>|<DIGIT>)*
> >   >
> >
> >
> > Should I remove "_" and recompile the source code?
> >
> > Sincerely,
> >
> >
> > Anh Ngo
> >
> > -----Original Message-----
> > From: Daniel Naber [mailto:lucenelist2005@danielnaber.de]
> > Sent: Friday, July 21, 2006 2:49 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: StandardAnalyzer question
> >
> > On Freitag 21 Juli 2006 16:16, Ngo, Anh (ISS Southfield) wrote:
> >
> > > The lucene 2.0.0 StandardAnalyzer does treat the "_"(underscore) as
> a
> > > token. Is there a way I can make StandardAnalyzer don't tokenize for
> > > "_" or any given characters?
> >
> > You need to add "_" to the #LETTER definition in StandardTokenizer.jj,
> > then
> > rebuild StandardTokenizer.java using the appropriate and task.
> >
> > Regards
> > Daniel
> >
> > --
> > http://www.danielnaber.de
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

------=_Part_9_20444770.1153512568127--