lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Search while indexing
Date Sat, 07 Mar 2009 14:58:00 GMT
First, you'll probably want to search the user list archive for this issue,
as
it's been discussed and you'll find more information than I can remember
off the top of my head. That said:

1> changes to an index are not visible until you reopen the reader. You
     probably have to flush the writer in the meantime. And this will
    be costly to do for every document.

2> How do you identify duplicates? If it's a short enough signature,
      you could consider keeping an in-memory list and check that
      while indexing. If you needed to update your index you could
      simply use TermEnum/TermDocs to read all the values into
      memory before adding to it.

3> You could consider using some kind of calculated signature of
     the whole file for your key, but that may not suit your app.

Best
Erick



On Sat, Mar 7, 2009 at 12:21 AM, sonfon <sonfon@gmail.com> wrote:

> Dear All,
>    Now, I'm considering to build index for my application with lucene.
> However, as the document sources I'm going to index has many duplications,
> so before adding a document to an IndexWriter, I hope search in the index
> database first to see if a same document copy has already been added. I used
> IndexSearcher to search the same Dir while IndexWriter writing to it.
> However, it seem that IndexSearcher returned no result though I'm sure there
> are duplicate copies indexed already. And after the indexing procedure, I
> can get the search results, so I'm sure I didn't write the wrong code.
> Anyone could offer some help? Some example codes are appreciated.
>    Best
> Wishes.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message