Return-Path: X-Original-To: apmail-hbase-dev-archive@www.apache.org Delivered-To: apmail-hbase-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 04A6618B30 for ; Tue, 16 Feb 2016 14:33:20 +0000 (UTC) Received: (qmail 82480 invoked by uid 500); 16 Feb 2016 14:33:18 -0000 Delivered-To: apmail-hbase-dev-archive@hbase.apache.org Received: (qmail 82351 invoked by uid 500); 16 Feb 2016 14:33:18 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 81951 invoked by uid 99); 16 Feb 2016 14:33:18 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Feb 2016 14:33:18 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 24A1B2C1F5C for ; Tue, 16 Feb 2016 14:33:18 +0000 (UTC) Date: Tue, 16 Feb 2016 14:33:18 +0000 (UTC) From: "stack (JIRA)" To: dev@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (HBASE-15275) Stuck thread while pulling data MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-15275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-15275. --------------------------- Resolution: Duplicate Resolving as duplicate of HBASE-15140 Please try it and then if it does not work for you, then open an issue. Thanks. > Stuck thread while pulling data > ------------------------------- > > Key: HBASE-15275 > URL: https://issues.apache.org/jira/browse/HBASE-15275 > Project: HBase > Issue Type: Bug > Affects Versions: 1.1.2 > Environment: CENT OS > Reporter: Arulanand Dayalan > Priority: Blocker > > I am trying to put data into an hbase table. It works for some time and gets stuck. Please find the stack trace. Is this related to HBASE-15140 and can i apply the same fix. > 2016-02-16 16:13:12 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode): > "hconnection-0x2c79c7b5-metaLookup-shared--pool2-t17" daemon prio=10 tid=0x00007f6ffc03b800 nid=0x2468 in Object.wait() [0x00007f7022e60000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007d73f5570> (a java.util.concurrent.atomic.AtomicBoolean) > at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:168) > - locked <0x00000007d73f5570> (a java.util.concurrent.atomic.AtomicBoolean) > at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Attach Listener" daemon prio=10 tid=0x00007f701806a000 nid=0x242f waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "hconnection-0x2c79c7b5-metaLookup-shared--pool2-t16" daemon prio=10 tid=0x00007f6ffc03a000 nid=0x2421 waiting on condition [0x00007f7022b59000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007fc0009c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "ConsumerFetcherThread-metrics-consumer_blb44cehhdp064-1455618399743-42fd2113-0-0" prio=10 tid=0x00007f701c6a5000 nid=0x22aa waiting on condition [0x00007f70349e4000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007fd0bfa60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349) > at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:60) > at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:49) > at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:128) > at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:109) > at scala.collection.immutable.Map$Map1.foreach(Map.scala:109) > at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:109) > at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109) > at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109) > at kafka.utils.Utils$.inLock(Utils.scala:535) > at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:108) > at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) > "metrics-consumer_blb44cehhdp064-1455618399743-42fd2113-leader-finder-thread" prio=10 tid=0x00007f701408a000 nid=0x22a9 waiting on condition [0x00007f7023264000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007fd14dd70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:61) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) > "kafkaProducer hbase - Thread-EventThread" daemon prio=10 tid=0x00007f6ffeaa8000 nid=0x2006 waiting on condition [0x00007f7022f61000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007fd9d64a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) > "kafkaProducer hbase - Thread-SendThread(hadoopmasterbkup:2181)" daemon prio=10 tid=0x00007f6fff603000 nid=0x2005 runnable [0x00007f7023062000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007fd9da0c0> (a sun.nio.ch.Util$2) > - locked <0x00000007fd9da0b0> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000007fd9d9c48> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) > "kafkaProducer hbase - Thread" prio=10 tid=0x00007f6ffdc11000 nid=0x1fc9 in Object.wait() [0x00007f7023162000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007d73f5450> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;) > at java.lang.Object.wait(Object.java:503) > at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148) > - locked <0x00000007d73f5450> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;) > at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188) > at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59) > at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) > at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:211) > at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1249) > at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1155) > at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370) > at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321) > at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206) > at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183) > - locked <0x00000007fd9da940> (a org.apache.hadoop.hbase.client.BufferedMutatorImpl) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1449) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1052) > at com.test.ceh.management.hbase.dao.impl.HbaseBaseDao.putAll(HbaseBaseDao.java:142) > at com.test.ceh.management.hbase.dao.impl.MetricsDao.putAll(MetricsDao.java:48) > at com.test.ceh.management.hbase.consumer.HbaseMetricsConsumer.consume(HbaseMetricsConsumer.java:30) > at com.test.ceh.management.runner.AdvancedRunnerImpl$ProcessorRunnable.invoke(AdvancedRunnerImpl.java:192) > at com.test.ceh.management.runner.AdvancedRunnerImpl$ProcessorRunnable.run(AdvancedRunnerImpl.java:158) > at java.lang.Thread.run(Thread.java:745) > "metrics-consumer_blb44cehhdp064-1455618399743-42fd2113_watcher_executor" prio=10 tid=0x00007f6ffdf57800 nid=0x1fc7 waiting on condition [0x00007f7023365000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007fd14ba60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2176) > at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:544) > "kafka-consumer-scheduler-0" daemon prio=10 tid=0x00007f6ffc8c8800 nid=0x1fc6 waiting on condition [0x00007f7023466000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007fd14ccc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "main-EventThread" daemon prio=10 tid=0x00007f6fffa83000 nid=0x1fc5 waiting on condition [0x00007f7023567000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007fd122e58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) > "main-SendThread(zookeeper1:2181)" daemon prio=10 tid=0x00007f6ffefd1000 nid=0x1fc4 runnable [0x00007f7023668000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007fd122800> (a sun.nio.ch.Util$2) > - locked <0x00000007fd1227f0> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000007fd122390> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) > "ZkClient-EventThread-29-zookeeper1:2181,hbasemasterbkup:2181,hadoopmasterbkup:2181" daemon prio=10 tid=0x00007f6fec309000 nid=0x1fc3 waiting on condition [0x00007f702386a000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007fd14d9d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:67) > "metrics-meter-tick-thread-2" daemon prio=10 tid=0x00007f6ffe5c6800 nid=0x1fc2 waiting on condition [0x00007f7023769000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007fc8945e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "metrics-meter-tick-thread-1" daemon prio=10 tid=0x00007f6ffe9ff000 nid=0x1fc1 waiting on condition [0x00007f7023b6c000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007fc8945e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "nagios kafka - Thread" prio=10 tid=0x00007f6ffe8fa000 nid=0x1fba waiting on condition [0x00007f7023c6d000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at java.lang.Thread.sleep(Thread.java:340) > at com.test.ceh.management.runner.AdvancedRunnerImpl$ProcessorRunnable.run(AdvancedRunnerImpl.java:176) > at java.lang.Thread.run(Thread.java:745) > "kafka-producer-network-thread | kafkaclient-1" daemon prio=10 tid=0x00007f6ffe8ec800 nid=0x1fb9 runnable [0x00007f7023d6e000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007fd50e0e0> (a sun.nio.ch.Util$2) > - locked <0x00000007fd50e0d0> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000007fd50dbf0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at org.apache.kafka.common.network.Selector.select(Selector.java:328) > at org.apache.kafka.common.network.Selector.poll(Selector.java:218) > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) > at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) > at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) > at java.lang.Thread.run(Thread.java:745) > "Curator-Framework-0" daemon prio=10 tid=0x00007f6ffddf8800 nid=0x1fb7 waiting on condition [0x00007f70341bc000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007fd48f520> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.DelayQueue.take(DelayQueue.java:209) > at java.util.concurrent.DelayQueue.take(DelayQueue.java:68) > at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780) > at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) > at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "main-EventThread" daemon prio=10 tid=0x00007f6ffd350000 nid=0x1fb6 waiting on condition [0x00007f70342bd000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007fd48ed58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) > "main-SendThread(zookeeper1:2181)" daemon prio=10 tid=0x00007f6ffd3ba000 nid=0x1fb5 runnable [0x00007f70343be000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007fd48e6f0> (a sun.nio.ch.Util$2) > - locked <0x00000007fd48e6e0> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000007fd48e280> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) > "Curator-ConnectionStateManager-0" daemon prio=10 tid=0x00007f6ffd1b6800 nid=0x1fb4 waiting on condition [0x00007f70344bf000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007fd4b7c18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374) > at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245) > at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43) > at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "DestroyJavaVM" prio=10 tid=0x00007f7038009000 nid=0x1fa0 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Service Thread" daemon prio=10 tid=0x00007f7038121800 nid=0x1fa9 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" daemon prio=10 tid=0x00007f703811f800 nid=0x1fa8 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" daemon prio=10 tid=0x00007f703811c800 nid=0x1fa7 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" daemon prio=10 tid=0x00007f703811a000 nid=0x1fa6 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Finalizer" daemon prio=10 tid=0x00007f70380f1800 nid=0x1fa5 in Object.wait() [0x00007f703c508000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007fc08dbe0> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > - locked <0x00000007fc08dbe0> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Reference Handler" daemon prio=10 tid=0x00007f70380ef800 nid=0x1fa4 in Object.wait() [0x00007f703c609000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007fc05af88> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:503) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) > - locked <0x00000007fc05af88> (a java.lang.ref.Reference$Lock) > "VM Thread" prio=10 tid=0x00007f70380eb000 nid=0x1fa3 runnable > "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f703801e800 nid=0x1fa1 runnable > "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f7038020800 nid=0x1fa2 runnable > "VM Periodic Task Thread" prio=10 tid=0x00007f703812c800 nid=0x1faa waiting on condition > JNI global references: 147 -- This message was sent by Atlassian JIRA (v6.3.4#6332)