Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: domain of erickerickson@gmail.com
 designates 64.233.166.178 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=b7NbafOKVAUSJrFOF/hu2oa0socCe5Q2IscDdidazpsnIHGWfxEx7U7HfIZh5B6LPaDM0hXYsEvgivAgopKigK84uQT9vdLV3EzQ6SoiRMKQvOL655OykwZId3uCX7mS/2MAXtTisuHCgRccGp2N8AG4yfQck7kHSEk/lj5fIa4=
Message-ID: <359a92830608291046l4cd85347ice7e5f75af63ad72@mail.gmail.com>
Date: Tue, 29 Aug 2006 13:46:12 -0400
From: "Erick Erickson" <erickerickson@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: Installing a custom tokenizer
In-Reply-To: <d746a7a72a985ab1dacb0e1a6ae4ed16@as-st.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_13371_17300962.1156873572243"
References: <44F3F64B.8030709@sirma.bg>
	 <d746a7a72a985ab1dacb0e1a6ae4ed16@as-st.com>

------=_Part_13371_17300962.1156873572243
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

I'm in a real rush here, so pardon my brevity, but..... one of the
constructors for IndexWriter takes an Analyzer as a parameter, which can be
a PerFieldAnalyzerWrapper. That, if I understand your issue, should fix you
right up.

Same kind of thing for a Query.

Erick

On 8/29/06, Bill Taylor <wataylor@as-st.com> wrote:
>
> I am indexing documents which are filled with government jargon.  As
> one would expect, the standard tokenizer has problems with
> governmenteese.
>
> In particular, the documents use words such as 310N-P-Q as references
> to other documents.  The standard tokenizer breaks this "word" at the
> dashes so that I can find P or Q but not the entire token.
>
> I know how to write a new tokenizer.  I would like hints on how to
> install it and get my indexing system to use it.  I don't want to
> modify the standard .jar file.  What I think I want to do is set up my
> indexing operation to use the WhitespaceTokenizer instead of the normal
> one, but I am unsure how to do this.
>
> I know that the IndexTask has a setAnalyzer method.  The document
> formats are rather complicated and I need special code to isolate the
> text strings which should be indexed.   My file analyzer isolates the
> string I want to index, then does
>
> doc.add(new Field(DocFormatters.CONTENT_FIELD, <string from the file>,
> Field.Store.YES, Field.index.TOKENIZED));
>
> I suspect that my issue is getting the Field constructor to use a
> different tokenizer.  Can anyone help?
>
> Thanks.
>
> Bill Taylor
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

------=_Part_13371_17300962.1156873572243--