Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (herse.apache.org: domain of erickerickson@gmail.com
 designates 66.249.92.168 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=gJ+xEkxT3U6wR3DLRBbylt8XK3Mp6RlGNY0aNnrmQtHWVMo1Qw9cXza+HK1vg+zTsiAgttlUau9L3aJ49/mKZ/8TR0NPsbMWLmY3l+UBf5u1nKbFhTQVHy8JzxhzQBtF/oR6q6soNh0XI7fIqkgZhzzOpIvpaT14fxpzxx9EqFc=
Message-ID: <359a92830704020632w6a8d4178w259ccdbb00070bf1@mail.gmail.com>
Date: Mon, 2 Apr 2007 09:32:25 -0400
From: "Erick Erickson" <erickerickson@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: Searches fail while indexwriter is open
In-Reply-To: <9789072.post@talk.nabble.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_108772_9397154.1175520745299"
References: <9789072.post@talk.nabble.com>

------=_Part_108772_9397154.1175520745299
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Yes, you can search while index writes are taking place, but....

When you open an index reader, it essentially takes a snapshot
of the index and further modifications of the index are not visible to
that searcher as long as it's open. You must close and re-open the
reader (and associated searchers) to see your changes. How this
interacts with index writer flushing is...er...well I don't exactly know,
but this could well be an issue...

I wonder if this is what you're seeing. In broad terms, the base
question is how quickly you need to see changes in the index reflected
in your search results.

I suspect that the 10 file thing is a red herring, what do you have your
indexwriter parameters set at? Especially maxbuffereddocs (which has
a default value, perhaps not coincidentally, of 10).......

Lucene 2.1 has an IndexWriter.flush() method that could help......

Erick

On 4/2/07, baronDodd <campbaron@hotmail.com> wrote:
>
>
> I am currently writing a Lucene application and having a huge headache
> with
> concurrency.
>
> My requirements are that each time a file is indexed a search on its path
> is
> performed to see if an update (delete then re-index) is required. If a
> document with the same path exists then an IndexReader deletes the doc and
> then a writer reindexes the fiel. Sadly due to requirements the deletes
> and
> indexes can not be batch performed and I am constantly opening and closing
> the IndexReader and IndexWriter between multiple threads. Everything has
> been working fine and seems thread safe apart from this:
>
> If I index a test batch of 10 files and then once again a few minutes
> later
> repeat the operation on the same files then all 10 are updated ok. However
> when I perform the same test with more than about 10 files then my
> searches
> fail to find about 25% of the already existing files and I end up with
> duplicate entries in the index. I have managed to fix this by closing the
> indexWriter every time an update search is performed but this has taken
> performance to almost embarrasing levels! My understanding was that you
> could search a Lucene index with an IndexSearcher while any write
> operations
> are taking place? Is it possible that the search skips segments which are
> currently being written to?
> --
> View this message in context:
> http://www.nabble.com/Searches-fail-while-indexwriter-is-open-tf3505182.html#a9789072
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

------=_Part_108772_9397154.1175520745299--