Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 7 Jun 2017 06:51:18 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12955224.1459498800000.15247.1496818278394@Atlassian.JIRA>
In-Reply-To: <JIRA.12955224.1459498800000@Atlassian.JIRA>
References: <JIRA.12955224.1459498800000@Atlassian.JIRA> <JIRA.12955224.1459498800017@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-15576) Scanning cursor to prevent
 blocking long time on ResultScanner.next()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 07 Jun 2017 06:51:25 -0000


    [ https://issues.apache.org/jira/browse/HBASE-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040320#comment-16040320 ] 

Hudson commented on HBASE-15576:
--------------------------------

SUCCESS: Integrated in Jenkins build HBase-1.4 #765 (See [https://builds.apache.org/job/HBase-1.4/765/])
HBASE-15576 Scanning cursor to prevent blocking long time on (yangzhe1991: rev 381c89b5cc463b58010ab443de9e1c1d40d8bf98)
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientUtil.java
* (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Cursor.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
* (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
* (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestScannerCursor.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* (edit) hbase-protocol/src/main/protobuf/Client.proto
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallableWithReplicas.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScannerContext.java


> Scanning cursor to prevent blocking long time on ResultScanner.next()
> ---------------------------------------------------------------------
>
>                 Key: HBASE-15576
>                 URL: https://issues.apache.org/jira/browse/HBASE-15576
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Phil Yang
>            Assignee: Phil Yang
>             Fix For: 2.0.0, 1.4.0
>
>         Attachments: HBASE-15576.branch-1.v01.patch, HBASE-15576.branch-1.v01.patch, HBASE-15576.branch-1.v01.patch, HBASE-15576.branch-1.v02.patch, HBASE-15576.v01.patch, HBASE-15576.v02.patch, HBASE-15576.v03.patch, HBASE-15576.v03.patch, HBASE-15576.v04.patch, HBASE-15576.v04.patch, HBASE-15576.v05.patch, HBASE-15576.v06.patch, HBASE-15576.v07.patch
>
>
> After 1.1.0 released, we have partial and heartbeat protocol in scanning to prevent responding large data or timeout. Now for ResultScanner.next(), we may block for longer time larger than timeout settings to get a Result if the row is very large, or filter is sparse, or there are too many delete markers in files.
> However, in some scenes, we don't want it to be blocked for too long. For example, a web service which handles requests from mobile devices whose network is not stable and we can not set timeout too long(eg. only 5 seconds) between mobile and web service. This service will scan rows from HBase and return it to mobile devices. In this scene, the simplest way is to make the web service stateless. Apps in mobile devices will send several requests one by one to get the data until enough just like paging a list. In each request it will carry a start position which depends on the last result from web service. Different requests can be sent to different web service server because it is stateless.
> Therefore, the stateless web service need a cursor from HBase telling where we have scanned in RegionScanner when HBase client receives an empty heartbeat. And the service will return the cursor to mobile device although the response has no data. In next request we can start at the position of cursor, without the cursor we have to scan from last returned result and we may timeout forever. And of course even if the heartbeat message is not empty we can still use cursor to prevent re-scan the same rows/cells which has beed skipped.
> Obviously, we will give up consistency for scanning because even HBase client is also stateless, but it is acceptable in this scene. And maybe we can keep mvcc in cursor so we can get a consistent view?
> HBASE-13099 had some discussion, but it has no further progress by now.
> API:
> In Scan we need a new method setNeedCursorResult(true) to get the cursor row key when there is a RPC response but client can not return any Result. In this mode we will not block ResultScanner.next() longer than this timeout setting.
> {code}
> while (r = scanner.next() && r != null) {
>   if(r.isCursor()){
>   // scanning is not end, it is a cursor, save its row key and close scanner if you want, or
>   // just continue the loop to call next().
>   } else {
>   // just like before
>   }
> }
> // scanning is end
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)