hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ryan rawson (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
Date Sun, 14 Jun 2009 07:50:07 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719229#action_12719229
] 

ryan rawson commented on HBASE-1500:
------------------------------------

the regionserver logs say something like so:

2009-06-14 00:44:43,751 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.NullPointerException
2009-06-14 00:44:43,752 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60021,
call next(-1247002001233657007, 30) from 10.10.20.13:36528: error: java.io.IOExc
eption: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:832)
       at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:822)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889)
        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:97)
        at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:85)
        at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:78)
        at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:120)
        at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:66)
        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)        at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:951)
        at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:907)
        at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1082)   
    at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:56)
        at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:140)
        at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1716)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1878)
        ... 5 more
2009-06-14 00:44:45,757 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.IllegalArgumentException
2009-06-14 00:44:45,758 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60021,
call next(-1247002001233657007, 30) from 10.10.20.13:36528: error: java.io.IOExc
eption: java.lang.IllegalArgumentException
java.io.IOException: java.lang.IllegalArgumentException
        at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:832)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:822)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889)
        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
Caused by: java.lang.IllegalArgumentException
        at java.nio.Buffer.position(Buffer.java:218)        at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1073)
        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:56)
        at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:140)
        at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1716)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1878)
        ... 5 more
2009-06-14 00:44:47,762 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.NegativeArraySizeException
2009-06-14 00:44:47,763 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60021,
call next(-1247002001233657007, 30) from 10.10.20.13:36528: error: java.io.IOException: java.lang.NegativeArraySizeException
java.io.IOException: java.lang.NegativeArraySizeException
        at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:832)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:822)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889)
        at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
Caused by: java.lang.NegativeArraySizeException
        at org.apache.hadoop.hbase.KeyValue.getRow(KeyValue.java:890)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1708)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1878)
        ... 5 more

then we get the negative array size exceptions from then on out.  very strange.

this seems very related to 1503.  I wonder how we can fix up hfile to be less errorful.

> KeyValue$KeyComparator array overrun
> ------------------------------------
>
>                 Key: HBASE-1500
>                 URL: https://issues.apache.org/jira/browse/HBASE-1500
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Andrew Purtell
>            Assignee: ryan rawson
>            Priority: Blocker
>             Fix For: 0.20.0
>
>         Attachments: hbase.log.gz, test.log.gz, test.rb
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set.  User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
>         at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
>         at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
>         at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
>         at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
>         at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
>         at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
>         at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
>         at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
>         at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
>         at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
>         at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
>         at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
>         at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message