lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Wang <john.w...@gmail.com>
Subject Re: IndexWriter.deleteDocuments(Query query)
Date Wed, 01 Apr 2009 21:22:28 GMT
Hi Michael:

    1) Yes, we use TermDocs, exactly what IndexWriter.deleteDocuments(Term)
is doing under the cover.
    2) We iterate the docid->uid mapping, for each docid, get the
corresponding ui and check that to see if that is in the deleted set. If so,
add the docid to the list. There is no uid->docid lookup needed.

      However, in our sharded architecture, we partition by continuous uids,
in which case we keep both mappings since we know the range of the the uid.
In which case, uid->docid mapping is available.

-John

On Wed, Apr 1, 2009 at 11:27 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> On Wed, Apr 1, 2009 at 2:04 PM, John Wang <john.wang@gmail.com> wrote:
>
> > My test essentially this. I took out the reader.deleteDocuments call from
> > both scenarios. I took a index of 5m docs. a batch of 10000 randomly
> > generated uids.
> >
> > Compared the following scenarios:
> > 1)
> > * open index reader
> > * for each uid in the batch, find the corresponding docid and add to an
> > IntList.
> > *close reader
>
> How exactly do you find the corresponding docid?  TermDocs?
>
> > 2)
> > * open index reader
> > * load uid array from payload field
> > * iterate uid array, and check to see if uid is in deleted set, and add
> to
> > an IntList
>
> In this case, each doc has a dedicated field that only has a payload
> that stores the one uid for that doc?  But I'm confused how you then
> map from uid -> docID.  I must be missing something.
>
> > The datastructure holding deleted set is IntOpenHashSet from fastutil.
> >
> > 1) took about 3500 - 4500 ms
> > 2) took about 815 ms
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message