hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15576) Scanning cursor to prevent blocking long time on ResultScanner.next()
Date Wed, 31 May 2017 12:06:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16031056#comment-16031056
] 

Hadoop QA commented on HBASE-15576:
-----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 35s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} |
{color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s {color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 11s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 38s {color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m 13s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s {color}
| {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 2s {color} | {color:red}
hbase-protocol-shaded in master has 24 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s {color} |
{color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 46s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 37s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 37s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m 13s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 34s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 27m 10s {color}
| {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5
2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m 23s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 11s {color} |
{color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 16s {color} | {color:red}
hbase-client generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s {color} | {color:green}
hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s {color} | {color:green}
hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 27s {color} | {color:green}
hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 108m 14s {color} | {color:red}
hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 23s {color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 177m 10s {color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12870543/HBASE-15576.v05.patch
|
| JIRA Issue | HBASE-15576 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  hbaseanti  checkstyle
 compile  cc  hbaseprotoc  |
| uname | Linux e295d693912d 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c945d2b |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/7021/artifact/patchprocess/branch-findbugs-hbase-protocol-shaded-warnings.html
|
| javadoc | https://builds.apache.org/job/PreCommit-HBASE-Build/7021/artifact/patchprocess/diff-javadoc-javadoc-hbase-client.txt
|
| unit | https://builds.apache.org/job/PreCommit-HBASE-Build/7021/artifact/patchprocess/patch-unit-hbase-server.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/7021/testReport/ |
| modules | C: hbase-protocol hbase-protocol-shaded hbase-client hbase-server U: . |
| Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/7021/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Scanning cursor to prevent blocking long time on ResultScanner.next()
> ---------------------------------------------------------------------
>
>                 Key: HBASE-15576
>                 URL: https://issues.apache.org/jira/browse/HBASE-15576
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Phil Yang
>            Assignee: Phil Yang
>             Fix For: 2.0.0, 1.4.0
>
>         Attachments: HBASE-15576.v01.patch, HBASE-15576.v02.patch, HBASE-15576.v03.patch,
HBASE-15576.v03.patch, HBASE-15576.v04.patch, HBASE-15576.v04.patch, HBASE-15576.v05.patch
>
>
> After 1.1.0 released, we have partial and heartbeat protocol in scanning to prevent responding
large data or timeout. Now for ResultScanner.next(), we may block for longer time larger than
timeout settings to get a Result if the row is very large, or filter is sparse, or there are
too many delete markers in files.
> However, in some scenes, we don't want it to be blocked for too long. For example, a
web service which handles requests from mobile devices whose network is not stable and we
can not set timeout too long(eg. only 5 seconds) between mobile and web service. This service
will scan rows from HBase and return it to mobile devices. In this scene, the simplest way
is to make the web service stateless. Apps in mobile devices will send several requests one
by one to get the data until enough just like paging a list. In each request it will carry
a start position which depends on the last result from web service. Different requests can
be sent to different web service server because it is stateless.
> Therefore, the stateless web service need a cursor from HBase telling where we have scanned
in RegionScanner when HBase client receives an empty heartbeat. And the service will return
the cursor to mobile device although the response has no data. In next request we can start
at the position of cursor, without the cursor we have to scan from last returned result and
we may timeout forever. And of course even if the heartbeat message is not empty we can still
use cursor to prevent re-scan the same rows/cells which has beed skipped.
> Obviously, we will give up consistency for scanning because even HBase client is also
stateless, but it is acceptable in this scene. And maybe we can keep mvcc in cursor so we
can get a consistent view?
> HBASE-13099 had some discussion, but it has no further progress by now.
> API:
> In Scan we need a new method setNeedCursorResult(true) to get the cursor row key when
there is a RPC response but client can not return any Result. In this mode we will not block
ResultScanner.next() longer than this timeout setting.
> {code}
> while (r = scanner.next() && r != null) {
>   if(r.isCursor()){
>   // scanning is not end, it is a cursor, save its row key and close scanner if you want,
or
>   // just continue the loop to call next().
>   } else {
>   // just like before
>   }
> }
> // scanning is end
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message