hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhouyingchao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-13835) KeyValueHeap.current might be in heap when exception happens in pollRealKV
Date Tue, 16 Jun 2015 10:55:01 GMT

     [ https://issues.apache.org/jira/browse/HBASE-13835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

zhouyingchao updated HBASE-13835:
---------------------------------
    Attachment: HBASE-13835-002.patch

Add an unit test case for the patch.  With the test case, it turned out that the exception
in pollRealKV might also lead to leak of close. I changed the patch a little bit to handle
the situation too.

> KeyValueHeap.current might be in heap when exception happens in pollRealKV
> --------------------------------------------------------------------------
>
>                 Key: HBASE-13835
>                 URL: https://issues.apache.org/jira/browse/HBASE-13835
>             Project: HBase
>          Issue Type: Bug
>          Components: Scanners
>            Reporter: zhouyingchao
>            Assignee: zhouyingchao
>         Attachments: HBASE-13835-001.patch, HBASE-13835-002.patch
>
>
> In a 0.94 hbase cluster, we found a NPE with following stack:
> {code}
> Exception in thread "regionserver21600.leaseChecker" java.lang.NullPointerException
>         at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1530)
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:225)
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:201)
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:191)
>         at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641)
>         at java.util.PriorityQueue.siftDown(PriorityQueue.java:612)
>         at java.util.PriorityQueue.poll(PriorityQueue.java:523)
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:241)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:355)
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:237)
>         at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:4302)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.leaseExpired(HRegionServer.java:3033)
>         at org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:119)
>         at java.lang.Thread.run(Thread.java:662)
> {code}
> Before this NPE exception, there is an exception happens in pollRealKV, which we think
is the culprit of the NPE.
> {code}
> ERROR org.apache.hadoop.hbase.regionserver.HRegionServer:
> java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=....
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:180)
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:371)
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:366)
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:116)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:455)
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:154)
>         at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:4124)
>         at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:4196)
>         at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:4067)
>         at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:4057)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.internalNext(HRegionServer.java:2898)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2833)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2815)
>         at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Server.call(SecureRpcEngine.java:337)
>         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1583)
> {code}
> Simply put, if there is an exception happens in pollRealKV( ), the KeyValueHeap.current
might be in heap. Later on, when KeyValueHeap.close( ) is called, the current would be closed
firstly. However, since it might still be in the heap, it would either be closed again or
its peek() (which is null after it is closed) is called by the heap's poll().  Neither case
is expected.
> Although it is caught in 0.94, it is still in the trunk from the code. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message