Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (herse.apache.org: domain of erickerickson@gmail.com
 designates 64.233.182.189 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=rveCnSw30yP5PM7izpfHurk2w9lBG7XMcuOqa/y8wi/S0RRx9CkShO0dgKexFhlIUR2zRzMvNkfLSICHNb+2ewJpQhHKPOWpKjrqjgZryTlPIgCixApHDJLVqxm6IgAkNTRFnAviW7IC5RzQUUQEjd6K5zj6CQzuvxzMQVXqfdg=
Message-ID: <359a92830611041900u502378c0md801397394cbc203@mail.gmail.com>
Date: Sat, 4 Nov 2006 22:00:15 -0500
From: "Erick Erickson" <erickerickson@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: 2.0 and Tokenized versus UN_TOKENIZED
In-Reply-To: <386a4720611041419m2b029019ke93d57857fe68eb7@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_34323_5257502.1162695615355"
References: <386a4720611041419m2b029019ke93d57857fe68eb7@mail.gmail.com>

------=_Part_34323_5257502.1162695615355
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Two questions come to mind...

1> what analyzer are you using for the *query*? Is it possible that when you
query for city you're using a tokenizer that breaks up your city code?

2> what about case? I'll assume that you have tried to search one-word
cities, so how the stream is tokenized won't break the query places you
don't expect. But if you index Austin UN_TOKENZED, then search for it using,
say StandardAnalyzer, it'll look for austin and they won't match. This may
apply to Luke too. In Luke, you can choose a different analyzer
(WhitespaceAnalyzer comes to mind).

Hope this helps
Erick

On 11/4/06, James Rhodes <jrrhodes@gmail.com> wrote:
>
> I'm using the 2.0 branch and I've had issues with searching indexes where
> the fields aren't tokenized.
> For instance, my index consists of count,lastname,city,state and I used
> the
> following code to index it (the data is in a sql server db):
> *
>
> if*(count != 0) {
>
> doc.add(*new* Field("count", NumberUtils.*pad*(count),
> Field.*Store*.*YES*,
> Field.Index.*TOKENIZED*));
>
> }
>
> *if*(lastName != *null*) {
>
> doc.add(*new* Field("lastname", lastName, Field.Store.*YES*, Field.Index.*
> TOKENIZED*,Field.TermVector.*YES*));
>
> }
>
> *if*(city != *null*) {
>
> doc.add(*new* Field("city", city, Field.Store.*YES*, Field.Index.*UN_**
> TOKENIZED*));
>
> }
>
> *if*(state != *null*) {
>
> *doc*.add(*new* Field("*state*", state, Field.Store.*YES*, Field.Index.*
> TOKENIZED*));
>
> }
>
> *Using this code I can search by any field with my app EXCEPT city, though
> I
> see it in the index using Luke.  I also can't search for it using Luke.
> When
> I add Field.Index.TOKENIZED  to the city field, I can search by it fine.*
>
> *Is this normal behavior? This doesn't make sense to me. Tokenized should
> prevent me from searching unless I'm missing something. Any ideas?
> Thanks!*
>
> *B*
>
>

------=_Part_34323_5257502.1162695615355--