Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: domain of erickerickson@gmail.com
 designates 64.233.166.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=BtTXEJIXiNvE/NES+7nkjxN5Dg5+ylVWYN90U/vIGbO18lWG2ok+vrFmEQqbdI3JUvqSJw0uNPtbmlzZnMFJVhhUPuwkskhwAWxUtN+ZuJ3jcbZBNQJbvblO56LvsXWLHOJV9ZNXvPNHEEpMXjXWtA7evDY9mDZ/8+KQLE54buM=
Message-ID: <359a92830608291357x1355af16p2a1f129cbce12add@mail.gmail.com>
Date: Tue, 29 Aug 2006 16:57:44 -0400
From: "Erick Erickson" <erickerickson@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: Installing a custom tokenizer
In-Reply-To: <617d2b451b3f7d8e9530363d5b4f4474@as-st.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_17217_26657837.1156885064463"
References: <33432A11DBA32B4EACBF6E37C5671DFA04F4EF@mailhyd2.hyd.deshaw.com>
	 <cf49d06c1d912dfaef83efcb08c4c583@as-st.com>
	 <1156867987.44f4679350377@webmail.uu.se>
	 <Pine.LNX.4.58.0608291144410.3758@hal.rescomp.berkeley.edu>
	 <617d2b451b3f7d8e9530363d5b4f4474@as-st.com>

------=_Part_17217_26657837.1156885064463
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Tucked away in the contrib section of  Lucene (I'm using 2.0) there is....

org.apache.lucene.index.memory.PatternAnalyzer

which takes a regular expression as and tokenizes with it. Would that help?
Word of warning... the regex determines what is NOT a token, not what IS a
token (as I remember), which threw me for a bit.

Don't know if this is really useful, but it might work for you without as
much work...

Best
Erick@I'mNowBeyondMyCompetence.WhyDoTheyStillEmployMeHere?

On 8/29/06, Bill Taylor <wataylor@as-st.com> wrote:
>
>
> On Aug 29, 2006, at 2:47 PM, Chris Hostetter wrote:
>
> >
> > : Have a look at PerFieldAnalyzerWrapper:
> >
> > :
> > http://lucene.apache.org/java/docs/api/org/apache/lucene/analysis/
> > PerFieldAnalyzerWrapper.html
> >
> > ...which can be specified in the constructors for IndexWriter and
> > QueryParser.
>
> As I understand it, this allows me to specify a different analyzer for
> each field name.  My problem is that the standard analyzer will not
> work for my content field and I need to define a new one.  I need to
> make a modification to the StandardTokenizer so that a number does not
> need to have a digit in every other segment of a part number.
>
> For example, the StandardTokenizer breaks aa-bb-2 on the - between aa
> and bb because it demands that every other string between a - have a
> digit.
>
> I need to modify the .jj file for the Standard Tokenizer and get a new
> one, but I am confused by the javaCC documentation and do not know how
> to run it to get what I need.
>
> Thanks for the help.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

------=_Part_17217_26657837.1156885064463--