hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: checksum exception
Date Wed, 19 Dec 2012 00:26:08 GMT
>
file:/home/stas/pricex/hbase/webpage/80e5ed6af22f12d45eabae89537f8602/f/5763919690181742358

Are you running HBase on HDFS?

I've seen checksum errors like this using Hadoop's LocalFS
(file:///foo/bar/) before. There isn't anything HBase can do here, this is
a problem that is in the (old) Hadoop libraries.

Also, the hadoop-core-0.20-append-r1056497 version of Hadoop was rather
specific to Facebook and a select few deployments at a certain point in
time in HBase history. Pretty far back now. I doubt anyone in Hadoop is
willing or able to support this today. Likewise, HBase 0.90.x is two
releases back from our current release. You'll find the community is much
better able to support you if using something more up to date. These may be
the versions that ship with that release of Nutch+Gora you are using, but
maybe there is a more recent option?

On Tue, Dec 18, 2012 at 4:09 PM, Stanislav Orlenko
<orlenko.stas@gmail.com>wrote:

> In logs also I see exceptions like:
>
> 2012-12-19 01:40:45,320 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> java.lang.IllegalArgumentException: offset (65577) + length (2) exceed the
> capacity of the array: 65577
>         at
>
> org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:506)
>         at org.apache.hadoop.hbase.util.Bytes.toShort(Bytes.java:728)
>         at org.apache.hadoop.hbase.util.Bytes.toShort(Bytes.java:714)
>         at org.apache.hadoop.hbase.KeyValue.getRowLength(KeyValue.java:733)
>         at org.apache.hadoop.hbase.KeyValue.getRow(KeyValue.java:894)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.peekRow(HRegion.java:2523)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2454)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2425)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2442)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1863)
>         at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
> I am not sure about hadoop version, in libs directory I see
> hadoop-core-0.20-append-r1056497.jar
>
> We use hbase as a backend for Gora in Nutch 2. Gora 0.2 uses HBase 0.90.X.
> That is why we use hbase 0.90.6. By the way in nutch's lib directory I see
> hadoop-core-1.0.3.jar
>
> Strange thing. After reboot it works without the exception for some time.
> Although after some period of time the checksum exception appears again.
> For example after previous reboot it has worked for about 24 hours, but now
> I see this exception again
>
>
> On Sun, Dec 16, 2012 at 2:44 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > The exception came from FSInputChecker.verifySum().
> >
> > Can you check Namenode log around the time the error happened to find
> more
> > information ?
> > What hadoop version do you use ?
> >
> > BTW 0.90.6 is really old. I suggest upgrading to 0.92 or 0.94
> >
> > Cheers
> >
> > On Sat, Dec 15, 2012 at 4:13 PM, Stanislav Orlenko
> > <orlenko.stas@gmail.com>wrote:
> >
> > > Hello
> > > We use HBase 0.90.6 and have the problem:
> > >
> > > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> > contact
> > > region server marv.site:56463 for region
> > >
> > >
> >
> webpage,pl.allegro:http/stare-rozdzielacze-i2839591826.html,1355478823557.80e5ed6af22f12d45eabae89537f8602.,
> > > row 'pl.allegro:http/stare-rozdzielacze-i2839591826.html', but failed
> > after
> > > 10 attempts.
> > > Exceptions:
> > > java.io.IOException: java.io.IOException: Could not iterate
> > > StoreFileScanner[HFileScanner for reader
> > >
> > >
> >
> reader=file:/home/stas/pricex/hbase/webpage/80e5ed6af22f12d45eabae89537f8602/f/5763919690181742358,
> > > compression=none, inMemory=false,
> > >
> > >
> >
> firstKey=pl.allegro:http/stare-rozdzielacze-i2839591826.html/f:bas/1355311168483/Put,
> > >
> lastKey=pl.gumtree.lodz:http/p-TermsAndConditions/f:ts/1355360972551/Put,
> > > avgKeyLen=83, avgValueLen=2880, entries=64207, length=191013462,
> > >
> > >
> >
> cur=pl.allegro:http/swiat-mebli-komoda-srebrna-61980-k14-i2787252255.html/f:cnt/1355480654644/Put/vlen=62169]
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:89)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:82)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:262)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2469)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2425)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2442)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1863)
> > >         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > >         at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > >         at java.lang.reflect.Method.invoke(Method.java:597)
> > >         at
> > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> > >         at
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> > > Caused by: org.apache.hadoop.fs.ChecksumException: Checksum error:
> > >
> > >
> >
> file:/home/stas/pricex/hbase/webpage/80e5ed6af22f12d45eabae89537f8602/f/5763919690181742358
> > > at 28099584
> > >         at
> > > org.apache.hadoop.fs.FSInputChecker.verifySum(FSInputChecker.java:277)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:241)
> > >         at
> > > org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> > >         at
> > > org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> > >         at java.io.DataInputStream.read(DataInputStream.java:132)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:105)
> > >         at
> > java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> > >         at
> java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> > >         at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> > >         at
> > >
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094)
> > >         at
> > >
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> > >         at
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1276)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:87)
> > >         ... 12 more
> > >
> > > java.io.IOException: java.io.IOException:
> > > java.lang.IllegalArgumentException
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:997)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:986)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1887)
> > >         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > >         at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > >         at java.lang.reflect.Method.invoke(Method.java:597)
> > >         at
> > > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> > >         at
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> > > Caused by: java.lang.IllegalArgumentException
> > >         at java.nio.Buffer.position(Buffer.java:218)
> > >         at
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1266)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:87)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:82)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:262)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2469)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2425)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2442)
> > >         at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1863)
> > >         ... 5 more
> > >
> > > We do not use special configuration. Only hbase.rootdir and
> > > hbase.zookeeper.property.dataDir are specified in hbase-site.
> > > What can be a reason for such behavior?
> > >
> > > Thanks in advance
> > >
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message