lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jm <jmugur...@gmail.com>
Subject Re: trying to resolve error: after flush: fdx size mismatch
Date Thu, 15 Apr 2010 08:11:04 GMT
Hi Mike

I have a server side, exposes a webservice with operations (this is
simplified ignoring things non lucene related):
- addToIndex(doc, date)
- flushAllIndexes()

addToIndex() adds the doc to the index, the index is chosen based on
the date (simplified it's one index per day). To speed this up the
webservice keeps a cache of IndexWriters.  If the index is already
open it uses it, otherwise adds it to the cache (the index is created
if it does not exist). I have a listener on the cache expiration so
the index is closed when it is removed from the cache (the cache has
only 10 items).

flushAllIndexes() just gets all indexes open on the cache and closes them all.

The webservice can serve many requests at the same time, but all
operations dealing with lucene are synchonized. I guess this is most
likely a bug in my synching code etc so I have verified it many times.
Even trying to reproduce the issue by stopping threads in a debugger
worked fine, synchronized methods were working as expected (I
understand am probably  missing the case that triggers the issue).

The last time this happened there was a single thread (in the client)
calling addToIndex(doc, date) many times and then it called
flushAllIndexes() The next second I got the error. The client
developer assures me the call to flushAllIndexes() as in the same
thread as addToIndex(doc, date) (anyway, it should work even if it was
different threads of course).

yes, local filesystem under Vista Business sp1 32b.

Hope this helps. I am just now developing a client that pounds the
server with some threads calling the methods all the time to try to
reproduce the issue.

javier

On Thu, Apr 15, 2010 at 1:47 AM, Michael McCandless
<lucene@mikemccandless.com> wrote:
> Not good!
>
> Can you describe how your threads work with Lucene?
>
> Is this just a local filesystem (disk) under vista?
>
> Mike
>
> On Wed, Apr 14, 2010 at 7:41 AM, jm <jmuguruza@gmail.com> wrote:
>> Hi,
>>
>> I am trying to chase an issue in our code and it is being quite
>> difficult. We have seen two instances (see below) where we get the
>> same error. I have been trying to reproduce but it has been impossible
>> so far.I have several threads, some might be creating indices and
>> adding documents, others closing indices etc.  In the list this
>> message appear several times, in some cases it was a lucene bug (not
>> my case, I dont have millions of docs here) and in other it seems bad
>> multithreading. I have verified my multithreaded code for the 5th time
>> and all seems well synchronized.
>>
>> So I am asking if somebody can give me any hint, maybe the fact that
>> the first number is always a 1 means something??  I suspect the issue
>> is when a new index is just created maybe.
>>
>> java.lang.RuntimeException: after flush: fdx size mismatch: 1 docs vs
>> 103 length in bytes of _2.fdx file exists?=true
>> org.apache.lucene.index.StoredFieldsWriter.closeDocStore(StoredFieldsWriter.java:97)
>> org.apache.lucene.index.DocFieldProcessor.closeDocStore(DocFieldProcessor.java:50)
>> org.apache.lucene.index.DocumentsWriter.closeDocStore(DocumentsWriter.java:380)
>> org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581)
>> org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4294)
>> org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192)
>> org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183)
>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2190)
>> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2153)
>> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2117)
>>
>>
>> java.lang.RuntimeException: after flush: fdx size mismatch: 1 docs vs
>> 8457 length in bytes of _0.fdx file exists?=true ...
>>
>> My env is vista, jdk1.6, lucene 2.9.1
>> thanks in advance.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message