Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 02F19200BC2 for ; Thu, 17 Nov 2016 18:45:08 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id F2847160B0B; Thu, 17 Nov 2016 17:45:07 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A6E94160AD8 for ; Thu, 17 Nov 2016 18:45:06 +0100 (CET) Received: (qmail 98170 invoked by uid 500); 17 Nov 2016 17:45:05 -0000 Mailing-List: contact dev-help@apex.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@apex.apache.org Delivered-To: mailing list dev@apex.apache.org Received: (qmail 98159 invoked by uid 99); 17 Nov 2016 17:45:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Nov 2016 17:45:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 6FDD61A0703 for ; Thu, 17 Nov 2016 17:45:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -6.219 X-Spam-Level: X-Spam-Status: No, score=-6.219 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id u_KdfKMt0wOC for ; Thu, 17 Nov 2016 17:45:01 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id EC8B85F297 for ; Thu, 17 Nov 2016 17:44:59 +0000 (UTC) Received: (qmail 96674 invoked by uid 99); 17 Nov 2016 17:44:58 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Nov 2016 17:44:58 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id A1C4B2C0059 for ; Thu, 17 Nov 2016 17:44:58 +0000 (UTC) Date: Thu, 17 Nov 2016 17:44:58 +0000 (UTC) From: "Vlad Rozov (JIRA)" To: dev@apex.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (APEXCORE-505) setup and activate calls in operator block heartbeat loop in container MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 17 Nov 2016 17:45:08 -0000 [ https://issues.apache.org/jira/browse/APEXCORE-505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-505: -------------------------------- Fix Version/s: 3.5.0 > setup and activate calls in operator block heartbeat loop in container > ---------------------------------------------------------------------- > > Key: APEXCORE-505 > URL: https://issues.apache.org/jira/browse/APEXCORE-505 > Project: Apache Apex Core > Issue Type: Bug > Reporter: Ashwin Chandra Putta > Assignee: Sandesh > Fix For: 3.5.0 > > > The setup and activate calls in the operators block the StreamingContainer heartbeat loop to send heartbeats to application master. As a result, if activation/setup takes more than the heartbeat timeout for any given operator within a container, the app master ends up killing the container for heartbeat timeout even though container is active. > To test this: I created simple test application and added sleep for 40 seconds in setup or activate calls. The application master shows the heart beat timeout message and kills the container with operator. Please find the stack trace on the container while it was active as follows: > sleep in activate callback > =================== > {code} > Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode): > "Attach Listener" daemon prio=10 tid=0x00007f8060ac0000 nid=0x6d2f waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "1/randomGenerator:RandomNumberGenerator" prio=10 tid=0x00007f8060b9b800 nid=0x65a2 waiting on condition [0x00007f8050bc8000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at com.datatorrent.sample.MyOperator.activate(MyOperator.java:63) > at com.datatorrent.stram.engine.Node.activate(Node.java:619) > at com.datatorrent.stram.engine.GenericNode.activate(GenericNode.java:205) > at com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1336) > at com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:130) > at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1396) > "ProcessWideEventLoop" prio=10 tid=0x00007f8060afb800 nid=0x658b runnable [0x00007f8050cc9000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007d186a380> (a com.datatorrent.netlet.OptimizedEventLoop$SelectedSelectionKeySet) > - locked <0x00000007d185a4a0> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000007d185a070> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) > at com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:185) > at com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:157) > at com.datatorrent.netlet.DefaultEventLoop.run(DefaultEventLoop.java:156) > at java.lang.Thread.run(Thread.java:745) > "Dispatcher-0" daemon prio=10 tid=0x00007f8060ae8000 nid=0x6586 waiting on condition [0x00007f8050dca000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dc5fa118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSyncAsyncMessageBus.java:51) > at java.lang.Thread.run(Thread.java:745) > "org.apache.hadoop.hdfs.PeerCache@389ed39" daemon prio=10 tid=0x00007f8060abf800 nid=0x6577 waiting on condition [0x00007f8050fcc000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255) > at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) > at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) > at java.lang.Thread.run(Thread.java:745) > "IPC Parameter Sending Thread #0" daemon prio=10 tid=0x00007f8060a2f800 nid=0x6572 waiting on condition [0x00007f80510cd000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007d511c0c0> (a java.util.concurrent.SynchronousQueue$TransferStack) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Thread-2" daemon prio=10 tid=0x00007f80609dd000 nid=0x656e runnable [0x00007f80516f2000] > java.lang.Thread.State: RUNNABLE > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method) > at org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(DomainSocketWatcher.java:52) > at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:474) > at java.lang.Thread.run(Thread.java:745) > "process reaper" daemon prio=10 tid=0x00007f80607c3000 nid=0x655d waiting on condition [0x00007f80580df000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007d2697c90> (a java.util.concurrent.SynchronousQueue$TransferStack) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" daemon prio=10 tid=0x00007f80600bd800 nid=0x6554 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" daemon prio=10 tid=0x00007f80600bb000 nid=0x6553 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" daemon prio=10 tid=0x00007f80600b8000 nid=0x6552 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" daemon prio=10 tid=0x00007f80600b6800 nid=0x6551 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Finalizer" daemon prio=10 tid=0x00007f8060096800 nid=0x6550 in Object.wait() [0x00007f8052dec000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007d0005608> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > - locked <0x00000007d0005608> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Reference Handler" daemon prio=10 tid=0x00007f8060094800 nid=0x654f in Object.wait() [0x00007f8052eed000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007d0005190> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:503) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) > - locked <0x00000007d0005190> (a java.lang.ref.Reference$Lock) > "main" prio=10 tid=0x00007f8060012000 nid=0x6538 waiting on condition [0x00007f806487d000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007ded02ed0> (a java.util.concurrent.CountDownLatch$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236) > at com.datatorrent.stram.engine.StreamingContainer.activate(StreamingContainer.java:1474) > - locked <0x00000007dc538160> (a com.datatorrent.stram.engine.StreamingContainer) > at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:874) > - locked <0x00000007dc538160> (a com.datatorrent.stram.engine.StreamingContainer) > at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:814) > at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:707) > at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:313) > "VM Thread" prio=10 tid=0x00007f8060090800 nid=0x654e runnable > "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f8060028000 nid=0x653a runnable > "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f806002a000 nid=0x653b runnable > "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f806002c000 nid=0x653c runnable > "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f806002d800 nid=0x653d runnable > "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f806002f800 nid=0x653e runnable > "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f8060031800 nid=0x653f runnable > "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f8060033000 nid=0x6540 runnable > "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f8060035000 nid=0x6541 runnable > "GC task thread#8 (ParallelGC)" prio=10 tid=0x00007f8060037000 nid=0x6542 runnable > "GC task thread#9 (ParallelGC)" prio=10 tid=0x00007f8060039000 nid=0x6543 runnable > "GC task thread#10 (ParallelGC)" prio=10 tid=0x00007f806003a800 nid=0x6544 runnable > "GC task thread#11 (ParallelGC)" prio=10 tid=0x00007f806003c800 nid=0x6545 runnable > "GC task thread#12 (ParallelGC)" prio=10 tid=0x00007f806003e800 nid=0x6546 runnable > "GC task thread#13 (ParallelGC)" prio=10 tid=0x00007f8060040000 nid=0x6547 runnable > "GC task thread#14 (ParallelGC)" prio=10 tid=0x00007f8060042000 nid=0x6548 runnable > "GC task thread#15 (ParallelGC)" prio=10 tid=0x00007f8060044000 nid=0x6549 runnable > "GC task thread#16 (ParallelGC)" prio=10 tid=0x00007f8060045800 nid=0x654a runnable > "GC task thread#17 (ParallelGC)" prio=10 tid=0x00007f8060047800 nid=0x654b runnable > "VM Periodic Task Thread" prio=10 tid=0x00007f80600c8800 nid=0x6555 waiting on condition > JNI global references: 277 > {code} > sleep in setup callback > =================== > {code} > 2016-08-11 17:59:20 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode): > "Attach Listener" daemon prio=10 tid=0x00007fda00ab8000 nid=0x22ea waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "1/randomGenerator:RandomNumberGenerator" prio=10 tid=0x00007fda00c08800 nid=0x1ffc waiting on condition [0x00007fd9f06a2000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at com.datatorrent.sample.MyOperator.setup(MyOperator.java:64) > at com.datatorrent.sample.MyOperator.setup(MyOperator.java:25) > at com.datatorrent.stram.engine.Node.setup(Node.java:187) > at com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1309) > at com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:130) > at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1396) > "ProcessWideEventLoop" prio=10 tid=0x00007fda00af5800 nid=0x1fdb runnable [0x00007fd9f07a3000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007d186a6e8> (a com.datatorrent.netlet.OptimizedEventLoop$SelectedSelectionKeySet) > - locked <0x00000007d185a808> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000007d185a3d8> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) > at com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:185) > at com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:157) > at com.datatorrent.netlet.DefaultEventLoop.run(DefaultEventLoop.java:156) > at java.lang.Thread.run(Thread.java:745) > "Dispatcher-0" daemon prio=10 tid=0x00007fda00ae0000 nid=0x1fda waiting on condition [0x00007fd9f08a4000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dc5fc5b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSyncAsyncMessageBus.java:51) > at java.lang.Thread.run(Thread.java:745) > "org.apache.hadoop.hdfs.PeerCache@762eefed" daemon prio=10 tid=0x00007fda00ab7800 nid=0x1fd7 waiting on condition [0x00007fd9f0aa6000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255) > at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) > at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) > at java.lang.Thread.run(Thread.java:745) > "IPC Parameter Sending Thread #0" daemon prio=10 tid=0x00007fda00a27800 nid=0x1fd5 waiting on condition [0x00007fd9f0ba7000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007d511c610> (a java.util.concurrent.SynchronousQueue$TransferStack) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Thread-2" daemon prio=10 tid=0x00007fda009d4800 nid=0x1fd3 runnable [0x00007fd9f11cc000] > java.lang.Thread.State: RUNNABLE > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method) > at org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(DomainSocketWatcher.java:52) > at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:474) > at java.lang.Thread.run(Thread.java:745) > "process reaper" daemon prio=10 tid=0x00007fda007b3000 nid=0x1fc5 waiting on condition [0x00007fd9f8090000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007d2698208> (a java.util.concurrent.SynchronousQueue$TransferStack) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" daemon prio=10 tid=0x00007fda000bd800 nid=0x1fb2 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" daemon prio=10 tid=0x00007fda000bb000 nid=0x1fb1 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" daemon prio=10 tid=0x00007fda000b8000 nid=0x1fb0 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" daemon prio=10 tid=0x00007fda000b6800 nid=0x1faf runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Finalizer" daemon prio=10 tid=0x00007fda00096800 nid=0x1fad in Object.wait() [0x00007fd9f28c6000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007d0005608> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > - locked <0x00000007d0005608> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Reference Handler" daemon prio=10 tid=0x00007fda00094800 nid=0x1fac in Object.wait() [0x00007fd9f29c7000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007d0005190> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:503) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) > - locked <0x00000007d0005190> (a java.lang.ref.Reference$Lock) > "main" prio=10 tid=0x00007fda00012000 nid=0x1f98 waiting on condition [0x00007fda042f0000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007ded06778> (a java.util.concurrent.CountDownLatch$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236) > at com.datatorrent.stram.engine.StreamingContainer.activate(StreamingContainer.java:1474) > - locked <0x00000007dc53a5f8> (a com.datatorrent.stram.engine.StreamingContainer) > at com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:874) > - locked <0x00000007dc53a5f8> (a com.datatorrent.stram.engine.StreamingContainer) > at com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:814) > at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:707) > at com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:313) > "VM Thread" prio=10 tid=0x00007fda00090800 nid=0x1fab runnable > "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fda00028000 nid=0x1f99 runnable > "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fda0002a000 nid=0x1f9a runnable > "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007fda0002c000 nid=0x1f9b runnable > "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007fda0002d800 nid=0x1f9c runnable > "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007fda0002f800 nid=0x1f9d runnable > "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007fda00031800 nid=0x1f9e runnable > "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007fda00033000 nid=0x1f9f runnable > "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007fda00035000 nid=0x1fa0 runnable > "GC task thread#8 (ParallelGC)" prio=10 tid=0x00007fda00037000 nid=0x1fa1 runnable > "GC task thread#9 (ParallelGC)" prio=10 tid=0x00007fda00039000 nid=0x1fa2 runnable > "GC task thread#10 (ParallelGC)" prio=10 tid=0x00007fda0003a800 nid=0x1fa3 runnable > "GC task thread#11 (ParallelGC)" prio=10 tid=0x00007fda0003c800 nid=0x1fa4 runnable > "GC task thread#12 (ParallelGC)" prio=10 tid=0x00007fda0003e800 nid=0x1fa5 runnable > "GC task thread#13 (ParallelGC)" prio=10 tid=0x00007fda00040000 nid=0x1fa6 runnable > "GC task thread#14 (ParallelGC)" prio=10 tid=0x00007fda00042000 nid=0x1fa7 runnable > "GC task thread#15 (ParallelGC)" prio=10 tid=0x00007fda00044000 nid=0x1fa8 runnable > "GC task thread#16 (ParallelGC)" prio=10 tid=0x00007fda00045800 nid=0x1fa9 runnable > "GC task thread#17 (ParallelGC)" prio=10 tid=0x00007fda00047800 nid=0x1faa runnable > "VM Periodic Task Thread" prio=10 tid=0x00007fda000d0800 nid=0x1fb3 waiting on condition > JNI global references: 237 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)