Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 5 Nov 2013 18:45:18 +0000 (UTC)
From: "Sergey Shelukhin (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12675103.1382469300987.23590.1383677118985@arcas>
In-Reply-To: <JIRA.12675103.1382469300987@arcas>
References: <JIRA.12675103.1382469300987@arcas>
Subject: [jira] [Commented] (HBASE-9818) NPE in
 HFileBlock#AbstractFSReader#readAtOffset
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-9818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13814111#comment-13814111 ] 

Sergey Shelukhin commented on HBASE-9818:
-----------------------------------------

>From logs and code it seems like compactions are closing the stream via wrapper (via long chain of classes)
Beforehand, they are supposed to notify all scanners to rebuild heap, which is a synchronized method (on StoreScanner) and only then close (next/etc are also synchronized so it should all be properly sequenced). But somehow it's not happening I suspect.
Also in some cases stacks are during some initialization, not next(), so I haven't looked/not sure how that was supposed to be synched.
 I didn't have a lot of time, so just looked at the code. Looped testWritesWhileGetting 100 times and it never failed locally, which is sad.
Is there any chance/tool to find out when approx these failures started

> NPE in HFileBlock#AbstractFSReader#readAtOffset
> -----------------------------------------------
>
>                 Key: HBASE-9818
>                 URL: https://issues.apache.org/jira/browse/HBASE-9818
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Jimmy Xiang
>            Assignee: Ted Yu
>         Attachments: 9818-v2.txt
>
>
> HFileBlock#istream seems to be null.  I was wondering should we hide FSDataInputStreamWrapper#useHBaseChecksum.
> By the way, this happened when online schema change is enabled (encoding)
> {noformat}
> 2013-10-22 10:58:43,321 ERROR [RpcServer.handler=28,port=36020] regionserver.HRegionServer:
> java.lang.NullPointerException
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1200)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1436)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1318)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:359)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:503)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:553)
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:245)
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:166)
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:361)
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:336)
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:293)
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:258)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:603)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:476)
>         at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:129)
>         at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3546)
>         at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3616)
>         at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3494)
>         at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3485)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3079)
>         at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
>         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
>         at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
>         at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
>         at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
>         at java.lang.Thread.run(Thread.java:724)
> 2013-10-22 10:58:43,665 ERROR [RpcServer.handler=23,port=36020] regionserver.HRegionServer:
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 53438 But the nextCallSeq got from client: 53437; request=scanner_id: 1252577470624375060 number_of_rows: 100 close_scanner: false next_call_seq: 53437
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3030)
>         at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:27022)
>         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:1979)
>         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:90)
>         at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
>         at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
>         at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
>         at java.lang.Thread.run(Thread.java:724)
> {noformat}


--
This message was sent by Atlassian JIRA
(v6.1#6144)