lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Getting RuntimeException: after flush: fdx size mismatch while Indexing
Date Fri, 09 Dec 2011 21:12:52 GMT
Hmm... it looks like File.length() is somehow, sometimes lying, on
your NFS filesystem.

What's happening is Lucene is writing out a file, and it wrote 59540
bytes, closed the file (all with no exceptions), and then tried to
verify the length was 59540 but in fact the filesystem reported 32768
bytes as the length instead.

So it seems like your filesystem is acting flakey.... are there any
tunings/optimizations done on the file server, or dangerous options
when mounting?  Can you maybe instead index to a local filesystem and
then move the index to the NFS dir?

Mike McCandless

http://blog.mikemccandless.com

On Fri, Dec 9, 2011 at 2:22 PM, Jamir Shaikh <shaikhjamir@gmail.com> wrote:
> OS : RHEL 5.5 64 bit.
> Filesystem: NFS
>
> Thanks for the reply.
>
> Thanks,
> Jamir
>
> On Fri, Dec 9, 2011 at 10:22 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> Which OS/filesystem?
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Thu, Dec 8, 2011 at 9:46 PM, Jamir Shaikh <shaikhjamir@gmail.com>
>> wrote:
>> > I am using Lucene 3.5. I want to create around 30 million documents.
>> > While doing Indexing I am getting the following Exception:
>> >
>> > Caused by: java.lang.RuntimeException: after flush: fdx size mismatch:
>> 7442
>> > docs vs 32768 length in bytes of _ct.fdx file exists?=true
>> >
>> >                at
>> >
>> org.apache.lucene.index.StoredFieldsWriter.flush(StoredFieldsWriter.java:58)
>> >
>> >                at
>> >
>> org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:59)
>> >
>> >                at
>> > org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581)
>> >
>> >                at
>> > org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3623)
>> >
>> >                at
>> > org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3588)
>> >
>> >                at
>> > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2073)
>> >
>> >                at
>> > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2040)
>> >
>> >                at
>> >
>> com.cisco.ason.document.AbstractLuceneDocWriter.write(AbstractLuceneDocWriter.java:88)
>> >
>> >                ... 10 more
>> >
>> >
>> > I don't get this Exception consistently.
>> >
>> > I am not setting the maxBufferedDocs the default value is -1.
>> >
>> > So the flush is as per ramBufferSizeMB which is set to default 16.0 MB.
>> >
>> >
>> > Did anyone face this problem.
>> >
>> > I would appreciate any suggestions.
>> >
>> >
>> > --
>> > Thanks for your time,
>> > Jamir...
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message