lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From T Vinod Gupta <tvi...@readypulse.com>
Subject Re: index.merge.scheduler exception - java.io.IOException: Input/output error
Date Mon, 09 Jul 2012 22:17:44 GMT
thanks this was really helpful to understand whats going on..
i got these for 2 of my indexes -

WARNING: 29 broken segments (containing 385395 documents) detected
WARNING: would write new segments file, and 385395 documents would be lost,
if -fix were specified

WARNING: 29 broken segments (containing 385395 documents) detected
WARNING: would write new segments file, and 385395 documents would be lost,
if -fix were specified

so my only option is to run with -fix and live with the data loss right? no
other option right?
will running fsck help? sometimes my ls or less commands also hang for a
good 10 sec.. this somehow indicates that there is some corruption.

thanks

On Mon, Jul 9, 2012 at 6:27 AM, Erick Erickson <erickerickson@gmail.com>wrote:

> no, you can't delete those files, and you can't regenerate just those
> files,
> all the various segment files are necessary and intertwined...
>
> Consider using the CheckIndex facility, see:
> http://solr.pl/en/2011/01/17/checkindex-for-the-rescue/
>
> note, the CheckIndex class is contained in the lucene core jar....
>
> You can run it with the -fix option to repair (at, perhaps, the expense
> of loss of some documents) if you choose, but running it without
> that option first is probably a good idea..
>
> Best
> Erick
>
> On Mon, Jul 9, 2012 at 7:43 AM, T Vinod Gupta <tvinod@readypulse.com>
> wrote:
> > this is on local file system on amazon ec2 host. the file system was fine
> > until a week ago when the outage happened and there were probably some
> > system glitches. i have seen this issue since then.. sometimes regular
> > commands like less or ls hang for many seconds even though there is no
> > cpu/memory pressure on the machine.
> >
> > in my case, there are only 2 unique entries for which i see this error.
> one
> > for a .fdt file and one for a .tis file. is it possible to regenerate
> those
> > files somehow? if i delete those 2 files, will the entire index get
> > corrupted? im ok to live with some data loss if it makes it more stable
> and
> > performant.
> >
> > thanks
> >
> > On Mon, Jul 9, 2012 at 2:28 AM, Ian Lea <ian.lea@gmail.com> wrote:
> >
> >> Is this on a local or remote file system?  Is the file system itself
> >> OK?  Is something else messing with your lucene index at the same
> >> time?
> >>
> >>
> >> --
> >> Ian.
> >>
> >>
> >> On Sun, Jul 8, 2012 at 8:58 PM, T Vinod Gupta <tvinod@readypulse.com>
> >> wrote:
> >> > Hi,
> >> > My log files are showing the below exceptions almost at twice a minute
> >> > frequency. what is causing it and how can i fix it? I am not using
> lucene
> >> > directly but instead using elasticsearch (0.18.7 version). but since
> the
> >> > stack trace is all lucene, i am sending it to this mailing list.
> >> >
> >> > also, my queries are taking a long time to execute (sometimes take a
> >> > minute). could this be contributing to it somehow?
> >> >
> >> > [2012-07-08 19:44:19,887][WARN ][index.merge.scheduler    ] [<my node
> >> > name>] [twitter][4] failed to merge
> >> > java.io.IOException: Input/output error:
> >> > NIOFSIndexInput(path="/media/ephemeral0
> >> > /ES_data/elasticsearch/nodes/0/indices/twitter/4/index/_2h29k.tis")
> >> >         at
> >> > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(N
> >> > IOFSDirectory.java:180)
> >> >         at
> >> > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.
> >> > java:229)
> >> >         at
> >> >
> >>
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)
> >> >         at
> org.apache.lucene.store.DataInput.readVInt(DataInput.java:105)
> >> >         at
> >> >
> >>
> org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:197)
> >> >         at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:86)
> >> >         at
> >> > org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:133)
> >> >         at
> >> >
> org.apache.lucene.index.SegmentMergeInfo.next(SegmentMergeInfo.java:72)
> >> >         at
> >> >
> >>
> org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:546)
> >> >         at
> >> >
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:473)
> >> >         at
> >> > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:109)
> >> >         at
> >> > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4295)
> >> >         at
> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3940)
> >> >         at
> >> >
> >>
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
> >> >         at
> >> >
> >>
> org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:88)
> >> >         at
> >> >
> >>
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
> >> > Caused by: java.io.IOException: Input/output error
> >> >         at sun.nio.ch.FileDispatcher.pread0(Native Method)
> >> >         at sun.nio.ch.FileDispatcher.pread(FileDispatcher.java:49)
> >> >         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:248)
> >> >         at sun.nio.ch.IOUtil.read(IOUtil.java:224)
> >> >         at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:663)
> >> >         at
> >> >
> >>
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:162)
> >> >         ... 15 more
> >> >
> >> > thanks
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message