Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9E2C717E43 for ; Thu, 4 Jun 2015 01:56:38 +0000 (UTC) Received: (qmail 30118 invoked by uid 500); 4 Jun 2015 01:56:38 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 30064 invoked by uid 500); 4 Jun 2015 01:56:38 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 30051 invoked by uid 99); 4 Jun 2015 01:56:38 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Jun 2015 01:56:38 +0000 Date: Thu, 4 Jun 2015 01:56:38 +0000 (UTC) From: "zhouyingchao (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HBASE-13835) KeyValueHeap.current might be in heap when exception happens in pollRealKV MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-13835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhouyingchao updated HBASE-13835: --------------------------------- Status: Patch Available (was: Open) > KeyValueHeap.current might be in heap when exception happens in pollRealKV > -------------------------------------------------------------------------- > > Key: HBASE-13835 > URL: https://issues.apache.org/jira/browse/HBASE-13835 > Project: HBase > Issue Type: Bug > Components: Scanners > Reporter: zhouyingchao > Assignee: zhouyingchao > Attachments: HBASE-13835-001.patch > > > In a 0.94 hbase cluster, we found a NPE with following stack: > {code} > Exception in thread "regionserver21600.leaseChecker" java.lang.NullPointerException > at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1530) > at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:225) > at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:201) > at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:191) > at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641) > at java.util.PriorityQueue.siftDown(PriorityQueue.java:612) > at java.util.PriorityQueue.poll(PriorityQueue.java:523) > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:241) > at org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:355) > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:237) > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:4302) > at org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.leaseExpired(HRegionServer.java:3033) > at org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:119) > at java.lang.Thread.run(Thread.java:662) > {code} > Before this NPE exception, there is an exception happens in pollRealKV, which we think is the culprit of the NPE. > {code} > ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: > java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=.... > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:180) > at org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:371) > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:366) > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:116) > at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:455) > at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:154) > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:4124) > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:4196) > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:4067) > at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:4057) > at org.apache.hadoop.hbase.regionserver.HRegionServer.internalNext(HRegionServer.java:2898) > at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2833) > at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2815) > at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Server.call(SecureRpcEngine.java:337) > at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1583) > {code} > Simply put, if there is an exception happens in pollRealKV( ), the KeyValueHeap.current might be in heap. Later on, when KeyValueHeap.close( ) is called, the current would be closed firstly. However, since it might still be in the heap, it would either be closed again or its peek() (which is null after it is closed) is called by the heap's poll(). Neither case is expected. > Although it is caught in 0.94, it is still in the trunk from the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)