Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 3362D200C0D for ; Tue, 17 Jan 2017 07:14:37 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 323CC160B4D; Tue, 17 Jan 2017 06:14:37 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 327F7160B41 for ; Tue, 17 Jan 2017 07:14:36 +0100 (CET) Received: (qmail 36834 invoked by uid 500); 17 Jan 2017 06:14:35 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 36823 invoked by uid 99); 17 Jan 2017 06:14:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Jan 2017 06:14:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id B4FA1C15CA for ; Tue, 17 Jan 2017 06:14:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1.999 X-Spam-Level: X-Spam-Status: No, score=-1.999 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id K9Zw8vMKqe5R for ; Tue, 17 Jan 2017 06:14:33 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id F10A75F2C5 for ; Tue, 17 Jan 2017 06:14:32 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 12FEDE7BF1 for ; Tue, 17 Jan 2017 06:14:28 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 9063B25288 for ; Tue, 17 Jan 2017 06:14:26 +0000 (UTC) Date: Tue, 17 Jan 2017 06:14:26 +0000 (UTC) From: "Hadoop QA (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-17475) Stack overflow in AsyncProcess if retry too much MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 17 Jan 2017 06:14:37 -0000 [ https://issues.apache.org/jira/browse/HBASE-17475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825521#comment-15825521 ] Hadoop QA commented on HBASE-17475: ----------------------------------- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 14s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 8s {color} | {color:red} root in master failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s {color} | {color:green} master passed with JDK v1.8.0_111 {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 6s {color} | {color:red} hbase-client in master failed with JDK v1.7.0_80. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 27s {color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m 8s {color} | {color:red} hbase-client in master failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 5s {color} | {color:red} hbase-client in master failed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s {color} | {color:green} master passed with JDK v1.8.0_111 {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 7s {color} | {color:red} hbase-client in master failed with JDK v1.7.0_80. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 6s {color} | {color:red} hbase-client in the patch failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s {color} | {color:green} the patch passed with JDK v1.8.0_111 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 7s {color} | {color:red} hbase-client in the patch failed with JDK v1.7.0_80. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 7s {color} | {color:red} hbase-client in the patch failed with JDK v1.7.0_80. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m 6s {color} | {color:red} hbase-client in the patch failed. {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 6s {color} | {color:red} The patch causes 7 errors with Hadoop v2.4.0. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 12s {color} | {color:red} The patch causes 7 errors with Hadoop v2.4.1. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 17s {color} | {color:red} The patch causes 7 errors with Hadoop v2.5.0. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 23s {color} | {color:red} The patch causes 7 errors with Hadoop v2.5.1. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 29s {color} | {color:red} The patch causes 7 errors with Hadoop v2.5.2. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 35s {color} | {color:red} The patch causes 7 errors with Hadoop v2.6.1. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 40s {color} | {color:red} The patch causes 7 errors with Hadoop v2.6.2. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 46s {color} | {color:red} The patch causes 7 errors with Hadoop v2.6.3. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 52s {color} | {color:red} The patch causes 7 errors with Hadoop v2.7.1. {color} | | {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 6s {color} | {color:red} hbase-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 6s {color} | {color:red} hbase-client in the patch failed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s {color} | {color:green} the patch passed with JDK v1.8.0_111 {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 6s {color} | {color:red} hbase-client in the patch failed with JDK v1.7.0_80. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 5s {color} | {color:red} hbase-client in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 8s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 15m 44s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12847737/HBASE-17475.v2.patch | | JIRA Issue | HBASE-17475 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux ffd427a81d2a 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/hbase.sh | | git revision | master / 4cb09a4 | | Default Java | 1.7.0_80 | | Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-oracle:1.7.0_80 | | mvninstall | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/artifact/patchprocess/branch-mvninstall-root.txt | | compile | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/artifact/patchprocess/branch-compile-hbase-client-jdk1.7.0_80.txt | | mvneclipse | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/artifact/patchprocess/branch-mvneclipse-hbase-client.txt | | findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/artifact/patchprocess/branch-findbugs-hbase-client.txt | | javadoc | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/artifact/patchprocess/branch-javadoc-hbase-client-jdk1.7.0_80.txt | | mvninstall | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/artifact/patchprocess/patch-mvninstall-hbase-client.txt | | compile | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/artifact/patchprocess/patch-compile-hbase-client-jdk1.7.0_80.txt | | javac | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/artifact/patchprocess/patch-compile-hbase-client-jdk1.7.0_80.txt | | mvneclipse | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/artifact/patchprocess/patch-mvneclipse-hbase-client.txt | | hbaseprotoc | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/artifact/patchprocess/patch-hbaseprotoc-hbase-client.txt | | findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/artifact/patchprocess/patch-findbugs-hbase-client.txt | | javadoc | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/artifact/patchprocess/patch-javadoc-hbase-client-jdk1.7.0_80.txt | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/artifact/patchprocess/patch-unit-hbase-client.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/testReport/ | | modules | C: hbase-client U: hbase-client | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/5288/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Stack overflow in AsyncProcess if retry too much > ------------------------------------------------ > > Key: HBASE-17475 > URL: https://issues.apache.org/jira/browse/HBASE-17475 > Project: HBase > Issue Type: Bug > Components: API > Affects Versions: 2.0.0, 1.4.0 > Reporter: Allan Yang > Assignee: Allan Yang > Attachments: HBASE-17475-branch-1.patch, HBASE-17475-branch-1.v2.patch, HBASE-17475-branch-1.v3.patch, HBASE-17475.patch, HBASE-17475.v2.patch > > > In AsyncProcess, we resubmit the retry task in the same thread > {code} > // run all the runnables > for (Runnable runnable : runnables) { > if ((--actionsRemaining == 0) && reuseThread) { > runnable.run(); > } else { > try { > pool.submit(runnable); > } > ...... > {code} > But, if we retry too much time. soon, stack overflow will occur. This is very common in clusters with Phoenix. Phoenix need to write index table in the normal write path, retry will cause stack overflow exception. > {noformat} > "htable-pool19-t2" #582 daemon prio=5 os_prio=0 tid=0x0000000002687800 nid=0x4a96 waiting on condition [0x00007fe3f6301000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1174) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:977) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:886) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1181) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1321) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:575) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:729) > > Too long to present... > {noformat} > We need to control the {{reuseThread}}, if number of retry is too much, then don't reuse the same thread to prevent stack overflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)