hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruba Borthakur <dhr...@gmail.com>
Subject Re: SequenceFileLogReader uses a reflection hack resulting in runtime failures
Date Fri, 02 Dec 2011 07:16:47 GMT
what hadoop version are you using?


On Thu, Dec 1, 2011 at 11:12 PM, Mikhail Bautin <
bautin.mailing.lists@gmail.com> wrote:

> After fixing the getFileLength() method access bug, the error I'm seeing in
> my local multi-process cluster load test is different. Do we ever expect to
> see checksum errors on the local filesystem?
>
> 11/12/01 22:52:52 INFO wal.HLogSplitter: Splitting hlog:
> file:/tmp/hbase-mbautin/hbase/.logs/192.168.1.25
> ,54409,1322808663310-splitting/192.168.1.25
> %2C54409%2C1322808663310.1322808734995,
> length=25989120
> 11/12/01 22:52:52 INFO fs.FSInputChecker: Found checksum error: b[3627,
> 3627]=
> org.apache.hadoop.fs.ChecksumException: Checksum error:
> file:/tmp/hbase-mbautin/hbase/.logs/192.168.1.25
> ,54409,1322808663310-splitting/192.168.1.25
> %2C54409%2C1322808663310.1322808734995
> at 25689088
>        at
>
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:219)
>        at
>
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
>        at
> org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
>        at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>        at java.io.DataInputStream.read(DataInputStream.java:132)
>        at java.io.DataInputStream.readFully(DataInputStream.java:178)
>        at
>
> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
>        at
> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1937)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1837)
>        at
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1883)
>        at
>
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:203)
>        at
>
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:177)
>        at
>
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getNextLogLine(HLogSplitter.java:765)
>        at
>
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFileToTemp(HLogSplitter.java:402)
>         at
>
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFileToTemp(HLogSplitter.java:351)
>        at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:113)
>        at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:266)
>        at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:197)
>        at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:165)
>
> Thanks,
> --Mikhail
>
> On Thu, Dec 1, 2011 at 10:58 PM, Mikhail Bautin <
> bautin.mailing.lists@gmail.com> wrote:
>
> > @Stack: I am using hadoop-0.20.205.0 (the default Hadoop version from
> > pom.xml). There is a private getFileLength() method, but getMethod() does
> > not allow to retrieve it. We should use getDeclaredMethod() -- this
> appears
> > to work in my testing. I will include that fix in the HBaseClusterTest
> > diff. Not sure why no one saw this bug before.
> >
> > @Dhruba: I am running RestartMetaTest, which I am porting from 0.89-fb.
> > This is a test that starts a local HBase cluster as multiple processes
> (on
> > different ports), loads some data, and does a real kill -9 on the
> > regionserver serving meta. I saw this bug in the data loading part, not
> > because of killing the regionserver.
> >
> > Thanks,
> > --Mikhail
> >
> > On Thu, Dec 1, 2011 at 10:33 PM, Stack <stack@duboce.net> wrote:
> >
> >> On Thu, Dec 1, 2011 at 9:59 PM, Mikhail Bautin
> >> <bautin.mailing.lists@gmail.com> wrote:
> >> > 11/12/01 21:40:07 WARN wal.SequenceFileLogReader: Error while trying
> to
> >> get
> >> > accurate file length.  Truncation / data loss may occur if
> RegionServers
> >> > die.
> >> > java.lang.NoSuchMethodException:
> >> >
> >>
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.getFileLength()
> >>
> >> Your hadoop doesn't have this Mikhail?
> >>
> >> > Besides, even when it works, the reflection-based solution is probably
> >> > _much_ slower than straightforward method access. Should we create a
> >> Hadoop
> >> > patch to expose the appropriate API call and get rid of the reflection
> >> hack?
> >> >
> >>
> >> It would be the sensible thing to do; much more sensible than the
> >> reflection gymnastics we have going on here.
> >>
> >> St.Ack
> >>
> >
> >
>



-- 
Subscribe to my posts at http://www.facebook.com/dhruba

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message