Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of orlenko.stas@gmail.com
 designates 209.85.212.49 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CALte62ximMUM=XvOFZyeAsPNN+PVi43fY0ua1gdBXY-rxk5JKQ@mail.gmail.com>
References: 
 <CAEDwJT_Not6YPGoF2AxwjWDFpAYyYvPrbDWUzMvFC7Z47+ibQg@mail.gmail.com>
	<CALte62ximMUM=XvOFZyeAsPNN+PVi43fY0ua1gdBXY-rxk5JKQ@mail.gmail.com>
Date: Wed, 19 Dec 2012 02:09:39 +0200
Message-ID: 
 <CAEDwJT_pTROzkTF5bGtXUN7nBjwNhm5AkCpR6C1a8UeCqW_rMA@mail.gmail.com>
Subject: Re: checksum exception
From: Stanislav Orlenko <orlenko.stas@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=20cf3077692738940c04d1296e03

--20cf3077692738940c04d1296e03
Content-Type: text/plain; charset=ISO-8859-1

In logs also I see exceptions like:

2012-12-19 01:40:45,320 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.lang.IllegalArgumentException: offset (65577) + length (2) exceed the
capacity of the array: 65577
        at
org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:506)
        at org.apache.hadoop.hbase.util.Bytes.toShort(Bytes.java:728)
        at org.apache.hadoop.hbase.util.Bytes.toShort(Bytes.java:714)
        at org.apache.hadoop.hbase.KeyValue.getRowLength(KeyValue.java:733)
        at org.apache.hadoop.hbase.KeyValue.getRow(KeyValue.java:894)
        at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.peekRow(HRegion.java:2523)
        at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2454)
        at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2425)
        at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2442)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1863)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

I am not sure about hadoop version, in libs directory I see
hadoop-core-0.20-append-r1056497.jar

We use hbase as a backend for Gora in Nutch 2. Gora 0.2 uses HBase 0.90.X.
That is why we use hbase 0.90.6. By the way in nutch's lib directory I see
hadoop-core-1.0.3.jar

Strange thing. After reboot it works without the exception for some time.
Although after some period of time the checksum exception appears again.
For example after previous reboot it has worked for about 24 hours, but now
I see this exception again


On Sun, Dec 16, 2012 at 2:44 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> The exception came from FSInputChecker.verifySum().
>
> Can you check Namenode log around the time the error happened to find more
> information ?
> What hadoop version do you use ?
>
> BTW 0.90.6 is really old. I suggest upgrading to 0.92 or 0.94
>
> Cheers
>
> On Sat, Dec 15, 2012 at 4:13 PM, Stanislav Orlenko
> <orlenko.stas@gmail.com>wrote:
>
> > Hello
> > We use HBase 0.90.6 and have the problem:
> >
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> contact
> > region server marv.site:56463 for region
> >
> >
> webpage,pl.allegro:http/stare-rozdzielacze-i2839591826.html,1355478823557.80e5ed6af22f12d45eabae89537f8602.,
> > row 'pl.allegro:http/stare-rozdzielacze-i2839591826.html', but failed
> after
> > 10 attempts.
> > Exceptions:
> > java.io.IOException: java.io.IOException: Could not iterate
> > StoreFileScanner[HFileScanner for reader
> >
> >
> reader=file:/home/stas/pricex/hbase/webpage/80e5ed6af22f12d45eabae89537f8602/f/5763919690181742358,
> > compression=none, inMemory=false,
> >
> >
> firstKey=pl.allegro:http/stare-rozdzielacze-i2839591826.html/f:bas/1355311168483/Put,
> > lastKey=pl.gumtree.lodz:http/p-TermsAndConditions/f:ts/1355360972551/Put,
> > avgKeyLen=83, avgValueLen=2880, entries=64207, length=191013462,
> >
> >
> cur=pl.allegro:http/swiat-mebli-komoda-srebrna-61980-k14-i2787252255.html/f:cnt/1355480654644/Put/vlen=62169]
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:89)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:82)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:262)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2469)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2425)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2442)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1863)
> >         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> >         at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >         at java.lang.reflect.Method.invoke(Method.java:597)
> >         at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> >         at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> > Caused by: org.apache.hadoop.fs.ChecksumException: Checksum error:
> >
> >
> file:/home/stas/pricex/hbase/webpage/80e5ed6af22f12d45eabae89537f8602/f/5763919690181742358
> > at 28099584
> >         at
> > org.apache.hadoop.fs.FSInputChecker.verifySum(FSInputChecker.java:277)
> >         at
> >
> >
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:241)
> >         at
> > org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> >         at
> > org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> >         at java.io.DataInputStream.read(DataInputStream.java:132)
> >         at
> >
> >
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:105)
> >         at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> >         at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> >         at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:102)
> >         at
> > org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1094)
> >         at
> > org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
> >         at
> >
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1276)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:87)
> >         ... 12 more
> >
> > java.io.IOException: java.io.IOException:
> > java.lang.IllegalArgumentException
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:997)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:986)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1887)
> >         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> >         at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >         at java.lang.reflect.Method.invoke(Method.java:597)
> >         at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> >         at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> > Caused by: java.lang.IllegalArgumentException
> >         at java.nio.Buffer.position(Buffer.java:218)
> >         at
> >
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1266)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:87)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:82)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:262)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:2469)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2425)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2442)
> >         at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1863)
> >         ... 5 more
> >
> > We do not use special configuration. Only hbase.rootdir and
> > hbase.zookeeper.property.dataDir are specified in hbase-site.
> > What can be a reason for such behavior?
> >
> > Thanks in advance
> >
>

--20cf3077692738940c04d1296e03--