Return-Path: X-Original-To: apmail-curator-user-archive@minotaur.apache.org Delivered-To: apmail-curator-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9E4FB1084C for ; Fri, 8 Nov 2013 19:44:21 +0000 (UTC) Received: (qmail 37274 invoked by uid 500); 8 Nov 2013 19:44:21 -0000 Delivered-To: apmail-curator-user-archive@curator.apache.org Received: (qmail 37244 invoked by uid 500); 8 Nov 2013 19:44:21 -0000 Mailing-List: contact user-help@curator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@curator.apache.org Delivered-To: mailing list user@curator.apache.org Received: (qmail 37236 invoked by uid 99); 8 Nov 2013 19:44:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Nov 2013 19:44:21 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of metacret@gmail.com designates 74.125.83.47 as permitted sender) Received: from [74.125.83.47] (HELO mail-ee0-f47.google.com) (74.125.83.47) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Nov 2013 19:44:16 +0000 Received: by mail-ee0-f47.google.com with SMTP id c13so1219503eek.6 for ; Fri, 08 Nov 2013 11:43:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=gWtDJ3XuZHxlP8OCQrG40DPrXS3vYYhv5HCgu0jPw7g=; b=DlpeAz0uUQwrF5Ro8Bx4zsZHt5Tlnrqy6YZEX8EqyFSvxfW1t5XhUZkzwnf2q2XAeW iaA6XRxm8weffPiOylpYZywACl0T4pLzEWcKtWjQA1JoyanSOp/KbZ8N++iUk+twLx8A jwuqCn4a5ccpEt2GRKdIcwi28zX5t7Sso0vGJhFy+L8t4Qxm+cr9CdOMgR4BVa/TEXHC yO3s95TxWmfRA3k2mQHbgBSEKvJWR0iOPIKzMBxWkH+y3BqvpeYPBYpd9b/8a4DK4DAw GuIZ2eLJkoM7jIZPh/xsqgAfNA+93wElMO6pxJdDSwl1mbnq3+Lt3FBu56qX1eHRlpWy PccQ== MIME-Version: 1.0 X-Received: by 10.14.4.1 with SMTP id 1mr18936466eei.21.1383939834662; Fri, 08 Nov 2013 11:43:54 -0800 (PST) Received: by 10.223.201.132 with HTTP; Fri, 8 Nov 2013 11:43:54 -0800 (PST) Date: Fri, 8 Nov 2013 11:43:54 -0800 Message-ID: Subject: Blocked on InterProcessMutex From: "Bae, Jae Hyeon" To: user@curator.apache.org Content-Type: multipart/alternative; boundary=047d7b66f2853c53d604eaaf9a31 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b66f2853c53d604eaaf9a31 Content-Type: text/plain; charset=ISO-8859-1 Hi I got reported that unresponsive servers in production very frequently and the symptom is that all http threads are hung on ZK activity. It looks InternalBlockUntilConnectedOrTimeOut called from InterProcessMutex recipes. The followings are stack trace snippets. Do you have any clue what's wrong here? Thank you Best, Jae "http-0.0.0.0-7101-3" daemon prio=10 tid=0x00000000015c6000 nid=0x11ae waiting on condition [0x00007f91771ee000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000005ee0b7688> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:282) at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:296) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105) at org.apache.curator.framework.imps.CreateBuilderImpl.findProtectedNodeInForeground(CreateBuilderImpl.java:660) at org.apache.curator.framework.imps.CreateBuilderImpl.access$800(CreateBuilderImpl.java:42) at org.apache.curator.framework.imps.CreateBuilderImpl$10.call(CreateBuilderImpl.java:619) at org.apache.curator.framework.imps.CreateBuilderImpl$10.call(CreateBuilderImpl.java:610) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:606) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:429) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:409) at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:42) at org.apache.curator.framework.recipes.locks.LockInternals.attemptLock(LockInternals.java:224) at org.apache.curator.framework.recipes.locks.InterProcessMutex.internalLock(InterProcessMutex.java:221) at org.apache.curator.framework.recipes.locks.InterProcessMutex.acquire(InterProcessMutex.java:96) --047d7b66f2853c53d604eaaf9a31 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi

I got reported that unresponsive ser= vers in production very frequently and the symptom is that all http threads= are hung on ZK activity. It looks InternalBlockUntilConnectedOrTimeOut cal= led from InterProcessMutex recipes. The followings are stack trace snippets= . Do you have any clue what's wrong here?

Thank you
Best, Jae

=
"http-0.0.0= .0-7101-3" daemon prio=3D10 tid=3D0x00000000015c6000 nid=3D0x11ae wait= ing on condition [0x00007f91771ee000]
=A0 =A0java.lang= .Thread.State: TIMED_WAITING (parking)
at= sun.misc.Unsafe.park(Native Method)
- parking to wait for =A0<0x00000005ee0b768= 8> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.= LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared= Nanos(AbstractQueuedSynchronizer.java:1033)
at java.util.concurrent.locks.AbstractQueuedSy= nchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
at java.util.concurrent.CountDownLatch.await(C= ountDownLatch.java:282)
at org.apache.curator.CuratorZ= ookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient= .java:296)
at org.apache.curator.RetryLoo= p.callWithRetry(RetryLoop.java:105)
at or= g.apache.curator.framework.imps.CreateBuilderImpl.findProtectedNodeInForegr= ound(CreateBuilderImpl.java:660)
at org.apache.curator.framework.imps.CreateBui= lderImpl.access$800(CreateBuilderImpl.java:42)
at org.apache.curator.framewor= k.imps.CreateBuilderImpl$10.call(CreateBuilderImpl.java:619)
at org.apache.curator.framework.imps.CreateBuilderImpl$= 10.call(CreateBuilderImpl.java:610)
at org.apache.curator.RetryLoop.callWithRetry(= RetryLoop.java:107)
at org.apache.curator.framewor= k.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:606)
=
at org.apache.curator.framework.imps.CreateBui= lderImpl.forPath(CreateBuilderImpl.java:429)
at org.apache.curator.framework.imps.CreateBui= lderImpl.forPath(CreateBuilderImpl.java:409)
at org.apache.curator.framewor= k.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:42)
at org.apache.curator.framework.recipes.locks.LockIntern= als.attemptLock(LockInternals.java:224)
at org.apache.curator.framework.recipes.locks.= InterProcessMutex.internalLock(InterProcessMutex.java:221)
at org.apache.curator.framewor= k.recipes.locks.InterProcessMutex.acquire(InterProcessMutex.java:96)
<= /div>
--047d7b66f2853c53d604eaaf9a31--