Return-Path: Delivered-To: apmail-hadoop-hbase-dev-archive@minotaur.apache.org Received: (qmail 77648 invoked from network); 14 Jun 2009 07:50:19 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 14 Jun 2009 07:50:19 -0000 Received: (qmail 70771 invoked by uid 500); 14 Jun 2009 07:50:30 -0000 Delivered-To: apmail-hadoop-hbase-dev-archive@hadoop.apache.org Received: (qmail 70741 invoked by uid 500); 14 Jun 2009 07:50:30 -0000 Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hbase-dev@hadoop.apache.org Delivered-To: mailing list hbase-dev@hadoop.apache.org Received: (qmail 70731 invoked by uid 99); 14 Jun 2009 07:50:30 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Jun 2009 07:50:30 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Jun 2009 07:50:28 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id D22A7234C004 for ; Sun, 14 Jun 2009 00:50:07 -0700 (PDT) Message-ID: <1574050607.1244965807846.JavaMail.jira@brutus> Date: Sun, 14 Jun 2009 00:50:07 -0700 (PDT) From: "ryan rawson (JIRA)" To: hbase-dev@hadoop.apache.org Subject: [jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun In-Reply-To: <773638980.1244502127346.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719229#action_12719229 ] ryan rawson commented on HBASE-1500: ------------------------------------ the regionserver logs say something like so: 2009-06-14 00:44:43,751 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.NullPointerException 2009-06-14 00:44:43,752 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60021, call next(-1247002001233657007, 30) from 10.10.20.13:36528: error: java.io.IOExc eption: java.lang.NullPointerException java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:832) at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:822) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:97) at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:85) at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:78) at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:120) at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:66) at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74) at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:951) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:907) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1082) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:56) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:140) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1716) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1878) ... 5 more 2009-06-14 00:44:45,757 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.IllegalArgumentException 2009-06-14 00:44:45,758 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60021, call next(-1247002001233657007, 30) from 10.10.20.13:36528: error: java.io.IOExc eption: java.lang.IllegalArgumentException java.io.IOException: java.lang.IllegalArgumentException at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:832) at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:822) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913) Caused by: java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:218) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1073) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:56) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:140) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1716) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1878) ... 5 more 2009-06-14 00:44:47,762 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.NegativeArraySizeException 2009-06-14 00:44:47,763 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60021, call next(-1247002001233657007, 30) from 10.10.20.13:36528: error: java.io.IOException: java.lang.NegativeArraySizeException java.io.IOException: java.lang.NegativeArraySizeException at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:832) at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:822) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913) Caused by: java.lang.NegativeArraySizeException at org.apache.hadoop.hbase.KeyValue.getRow(KeyValue.java:890) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1708) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1878) ... 5 more then we get the negative array size exceptions from then on out. very strange. this seems very related to 1503. I wonder how we can fix up hfile to be less errorful. > KeyValue$KeyComparator array overrun > ------------------------------------ > > Key: HBASE-1500 > URL: https://issues.apache.org/jira/browse/HBASE-1500 > Project: Hadoop HBase > Issue Type: Bug > Affects Versions: 0.20.0 > Reporter: Andrew Purtell > Assignee: ryan rawson > Priority: Blocker > Fix For: 0.20.0 > > Attachments: hbase.log.gz, test.log.gz, test.rb > > > {code} > 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B > OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac > he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072 > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is > false > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA > 04/172.20.3.231:2181 > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan > nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231 > :2181] > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful > 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th > e arguments. Applications should implement Tool for the same. > 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may > not be found. See JobConf(Class) or JobConf#setJar(String). > 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B > OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac > he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54 > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA > 03/172.20.3.230:2181 > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan > nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230 > :2181] > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful > 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c > om:,01e33c601a7a9dd0ddb5c8427438f2f1 > Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32 > at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798) > at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j > ava:1760) > at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java: > 1696) > at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java: > 1755) > at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java: > 1687) > at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac > hedLocation(HConnectionManager.java:697) > at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate > RegionInMeta(HConnectionManager.java:541) > at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate > Region(HConnectionManager.java:525) > at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate > Region(HConnectionManager.java:488) > at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg > ionLocation(HConnectionManager.java:342) > at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1 > 91) > at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn > putFormatBase.java:296) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742) > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026) > at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source) > {code} -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.