Return-Path: Delivered-To: apmail-hadoop-hbase-dev-archive@minotaur.apache.org Received: (qmail 17916 invoked from network); 29 Dec 2009 17:42:57 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 29 Dec 2009 17:42:57 -0000 Received: (qmail 84547 invoked by uid 500); 29 Dec 2009 17:42:57 -0000 Delivered-To: apmail-hadoop-hbase-dev-archive@hadoop.apache.org Received: (qmail 84494 invoked by uid 500); 29 Dec 2009 17:42:56 -0000 Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hbase-dev@hadoop.apache.org Delivered-To: mailing list hbase-dev@hadoop.apache.org Received: (qmail 84484 invoked by uid 99); 29 Dec 2009 17:42:56 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Dec 2009 17:42:56 +0000 X-ASF-Spam-Status: No, hits=-10.5 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Dec 2009 17:42:50 +0000 Received: from brutus.apache.org (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id E4C89234C045 for ; Tue, 29 Dec 2009 09:42:29 -0800 (PST) Message-ID: <628795954.1262108549922.JavaMail.jira@brutus.apache.org> Date: Tue, 29 Dec 2009 17:42:29 +0000 (UTC) From: "Greg Lu (JIRA)" To: hbase-dev@hadoop.apache.org Subject: [jira] Created: (HBASE-2073) IllegalArgumentException causing regionserver failure MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 IllegalArgumentException causing regionserver failure ----------------------------------------------------- Key: HBASE-2073 URL: https://issues.apache.org/jira/browse/HBASE-2073 Project: Hadoop HBase Issue Type: Bug Components: regionserver Affects Versions: 0.20.2 Environment: Ubuntu 8.10, Java 1.6.0_10, HBase 0.20.2 Reporter: Greg Lu Attachments: hbase-hadoop-regionserver-factory05.lab.mtl.log After a regionserver went down last night, I checked its logs and found the following exception: 2009-12-29 00:17:27,663 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll /hbase/amsterdam_factory/.logs/factory05.lab.mtl,60020,1262042255724/hlog.dat.1262060247637, entries=1830, calcsize=22946017, filesize=22758899. New hlog /hbase/amsterdam_factory/.logs/factory05.lab.mtl,60020,1262042255724/hlog.dat.1262063847659 2009-12-29 00:34:36,210 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:218) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1114) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:189) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944) at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) 2009-12-29 00:34:36,214 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60020, call next(4170645244799815171, 1) from 192.168.1.108:53401: error: java.io.IOException: java.lang.IllegalArgumentException java.io.IOException: java.lang.IllegalArgumentException at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:869) at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:859) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1965) at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Caused by: java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:218) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1114) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:189) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944) ... 5 more Looks like this bug was encountered before at https://issues.apache.org/jira/browse/HBASE-1495 and spanned a few JIRAs. It's supposed to be resolved as of 0.20.0, but we're running 0.20.2 and it took down one of our regionservers. I'm also attaching more of the log. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.