Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of erickerickson@gmail.com
 designates 72.14.214.224 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=i/6Qzeg8vowboJi8+mzqI1PLeErUKzWpmJLx951uAcAGeXCNNjKxZN89Ix0lL75uqYZfxCkClFa23/FEsrj+XwJscIhaiOpu9KP9UlJhSF72OOR40seqQmAGclR97rMPaU+bv4FPNKUO2ysX+Fy5jUNm3XsPJMjr8NUn7XYsB+o=
Message-ID: <359a92830803120649t6349f937u9395523d76479d8c@mail.gmail.com>
Date: Wed, 12 Mar 2008 09:49:26 -0400
From: "Erick Erickson" <erickerickson@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: Unique Fields
In-Reply-To: <47D771A2.7070703@searchcapital.net>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_5689_22455366.1205329766750"
References: <47D69F7B.6020802@searchcapital.net>
	 <359a92830803111037u476bd57dxb7ea90972933d0ad@mail.gmail.com>
	 <47D771A2.7070703@searchcapital.net>

------=_Part_5689_22455366.1205329766750
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

So, you're tokenizing the title field? If so, I don't understand how you
expect
this to work. Would the title "this is one order" and "is one order this" be
considered
identical? Would capitalization matter? Punctuation? Throwing all the terms
of a title into a tokenized field and expecting some magic to keep
duplicates
is beyond the scope of Lucene, you'll have to roll some customized solution.

For instance, index your title UN_TOKENIZED in a duplicate field (after
applying
whatever massaging you want re: punctuation, spaces, etc.). Use
TermDocs/TermEnum
on that field to detect duplicates. You won't search on this field....

Or create a hash of the title and index *that* in a separate field and check
against
the hash with termenum/terndocs. Or.....

But no, there's no magic that makes Lucene DWIM (Do What I Mean)...

Best
Erick

On Wed, Mar 12, 2008 at 2:01 AM, Ion Badita <ion.badita@searchcapital.net>
wrote:

> The "problem" is that my unique field is a title, many terms per field.
> I want to make an index with titles and i don't want to have duplicates.
>
> John
>
>
> Erick Erickson wrote:
> > You can easily find whether a term is in the index with
> TermEnum/TermDocs
> > (I think TermEnum is all you really need).
> >
> > Except, you'll probably also have to keep an internal map of IDs added
> since
> > the searcher was opened and check against that too.
> >
> > Best
> > Erick
> >
> > On Tue, Mar 11, 2008 at 11:04 AM, Ion Badita <
> ion.badita@searchcapital.net>
> > wrote:
> >
> >
> >> Hi,
> >>
> >> I want to create an index with one unique field.
> >> Before inserting a document i must be sure that "unique field" is
> unique.
> >>
> >>
> >>
> >> John
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >>
> >
> >
>
>

------=_Part_5689_22455366.1205329766750--