Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 3C4E9200B7A for ; Mon, 22 Aug 2016 07:44:25 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 3AC88160AAA; Mon, 22 Aug 2016 05:44:25 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 5B522160AC0 for ; Mon, 22 Aug 2016 07:44:24 +0200 (CEST) Received: (qmail 42810 invoked by uid 500); 22 Aug 2016 05:44:23 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 42799 invoked by uid 99); 22 Aug 2016 05:44:23 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Aug 2016 05:44:23 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 53D092C0150 for ; Mon, 22 Aug 2016 05:44:23 +0000 (UTC) Date: Mon, 22 Aug 2016 05:44:23 +0000 (UTC) From: "Hadoop QA (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-15871) Memstore flush doesn't finish because of backwardseek() in memstore scanner. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 22 Aug 2016 05:44:25 -0000 [ https://issues.apache.org/jira/browse/HBASE-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15430086#comment-15430086 ] Hadoop QA commented on HBASE-15871: ----------------------------------- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 18s {color} | {color:green} branch-1.1 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s {color} | {color:green} branch-1.1 passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s {color} | {color:green} branch-1.1 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 33s {color} | {color:green} branch-1.1 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 22s {color} | {color:green} branch-1.1 passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 43s {color} | {color:red} hbase-server in branch-1.1 has 80 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 31s {color} | {color:red} hbase-server in branch-1.1 failed with JDK v1.8.0_101. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s {color} | {color:green} branch-1.1 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s {color} | {color:green} the patch passed with JDK v1.8.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 14 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 36s {color} | {color:red} The patch causes 11 errors with Hadoop v2.6.1. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 48s {color} | {color:red} The patch causes 11 errors with Hadoop v2.6.2. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 59s {color} | {color:red} The patch causes 11 errors with Hadoop v2.6.3. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 10s {color} | {color:red} The patch causes 11 errors with Hadoop v2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 58s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 23s {color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_101. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 29s {color} | {color:red} hbase-server-jdk1.7.0_101 with JDK v1.7.0_101 generated 1 new + 16 unchanged - 0 fixed = 17 total (was 16) {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 36s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 115m 22s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.mapreduce.TestHFileOutputFormat | | | hadoop.hbase.master.TestDistributedLogSplitting | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-22 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12809427/HBASE-15871.branch-1.1.003.patch | | JIRA Issue | HBASE-15871 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4e65f2702476 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/hbase.sh | | git revision | branch-1.1 / 1c92f4c | | Default Java | 1.7.0_101 | | Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 | | findbugs | v3.0.0 | | findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/3182/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html | | javadoc | https://builds.apache.org/job/PreCommit-HBASE-Build/3182/artifact/patchprocess/branch-javadoc-hbase-server-jdk1.8.0_101.txt | | whitespace | https://builds.apache.org/job/PreCommit-HBASE-Build/3182/artifact/patchprocess/whitespace-eol.txt | | javadoc | https://builds.apache.org/job/PreCommit-HBASE-Build/3182/artifact/patchprocess/patch-javadoc-hbase-server-jdk1.8.0_101.txt | | javadoc | https://builds.apache.org/job/PreCommit-HBASE-Build/3182/artifact/patchprocess/diff-javadoc-javadoc-hbase-server-jdk1.7.0_101.txt | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/3182/artifact/patchprocess/patch-unit-hbase-server.txt | | unit test logs | https://builds.apache.org/job/PreCommit-HBASE-Build/3182/artifact/patchprocess/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/3182/testReport/ | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/3182/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Memstore flush doesn't finish because of backwardseek() in memstore scanner. > ---------------------------------------------------------------------------- > > Key: HBASE-15871 > URL: https://issues.apache.org/jira/browse/HBASE-15871 > Project: HBase > Issue Type: Bug > Components: Scanners > Affects Versions: 1.1.2 > Reporter: Jeongdae Kim > Fix For: 1.1.2 > > Attachments: HBASE-15871.branch-1.1.001.patch, HBASE-15871.branch-1.1.002.patch, HBASE-15871.branch-1.1.003.patch, memstore_backwardSeek().PNG > > > Sometimes in our production hbase cluster, it takes a long time to finish memstore flush.( for about more than 30 minutes) > the reason is that a memstore flusher thread calls StoreScanner.updateReaders(), waits for acquiring a lock that store scanner holds in StoreScanner.next() and backwardseek() in memstore scanner runs for a long time. > I think that this condition could occur in reverse scan by the following process. > 1) create a reversed store scanner by requesting a reverse scan. > 2) flush a memstore in the same HStore. > 3) puts a lot of cells in memstore and memstore is almost full. > 4) call the reverse scanner.next() and re-create all scanners in this store because all scanners was already closed by 2)'s flush() and backwardseek() with store's lastTop for all new scanners. > 5) in this status, memstore is almost full by 2) and all cells in memstore have sequenceID greater than this scanner's readPoint because of 2)'s flush(). this condition causes searching all cells in memstore, and seekToPreviousRow() repeatly seach cells that are already searched if a row has one column. (described this in more detail in a attached file.) > 6) flush a memstore again in the same HStore, and wait until 4-5) process finished, to update store files in the same HStore after flusing. > I searched HBase jira. and found a similar issue. (HBASE-14497) but, HBASE-14497's fix can't solve this issue because that fix just changed recursive call to loop.(and already applied to our HBase version) -- This message was sent by Atlassian JIRA (v6.3.4#6332)