drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Khurram Faraaz (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-4330) Long running SQL query hangs once Foreman node is killed
Date Mon, 01 Feb 2016 11:56:39 GMT
Khurram Faraaz created DRILL-4330:
-------------------------------------

             Summary: Long running SQL query hangs once Foreman node is killed
                 Key: DRILL-4330
                 URL: https://issues.apache.org/jira/browse/DRILL-4330
             Project: Apache Drill
          Issue Type: Bug
          Components: Execution - Flow
    Affects Versions: 1.4.0
         Environment: 4 node cluster CentOS
            Reporter: Khurram Faraaz


Summary : Once Foreman node Drillbit is killed, long running query just hangs and no profile
information is written to Web UI. That long running query was issued from the Foreman node.

MapR Drill 1.4.0 GA
MapR FS 5.0.0 GA
JDK8
4 node CentOS cluster

./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
Issue a long running select query over JSON data
Immediately kill the Drillbit on Foreman node (ps -eaf | grep Drillbit), kill -9 PID

The long running query hangs on sqlline prompt, there are no messages/errors/Exceptions reported
on sqlline prompt.

On the Web UI there is no profile reported for the long running query that was running on
the Drillbit that was killed.

Question (1) : Why was there no profile reported/written on the Web UI for that long running
query ? In a real production scenario user will not know what query was under execution at
the point when Foreman went down. 

Question (2) : Why does the long running query not terminate, once the foreman was killed
? from the drillbit.log snippet we do not see any CANCELED/TERMINATED message for that query,
why ?

Snippet from drillbit.log on the foreman node. 
{noformat}
2016-02-01 10:59:20,917 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.drill.exec.work.foreman.Foreman
- Query text for query id 2950c576-b2d2-5bc3-e9b5-ff4414d088c0: select * from `twoKeyJsn.json`
2016-02-01 10:59:21,067 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.create() took 1 ms
2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,155 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.e.s.schedule.BlockMapBuilder
- Get block maps: Executed 1 out of 1 using 1 threads. Time: 90ms total, 90.891938ms avg,
90ms max.
2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.e.s.schedule.BlockMapBuilder
- Get block maps: Executed 1 out of 1 using 1 threads. Earliest start: 18.280000 μs, Latest
start: 18.280000 μs, Average start: 18.280000 μs .
2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State change requested AWAITING_ALLOCATION -->
RUNNING
2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO  o.a.d.e.w.f.FragmentStatusReporter
- 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State to report: RUNNING
{noformat}

Doing kill -3 PID on the non foreman node for the Drillbit process gives us stack trace in
drillbit.out
{noformat}
2016-02-01 11:03:31
Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):

"qtp801808302-129" #129 prio=5 os_prio=0 tid=0x00007f7ad8127000 nid=0xaad waiting on condition
[0x00007f7ab62fb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007b69100a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:513)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:569)
        at java.lang.Thread.run(Thread.java:745)

"DestroyJavaVM" #84 prio=5 os_prio=0 tid=0x00007f7b20012000 nid=0x74c0 waiting on condition
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"qtp801808302-81-acceptor-1-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #81 prio=5 os_prio=0
tid=0x00007f7b233e7000 nid=0x7a7d waiting for monitor entry [0x00007f7ab63fc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
        - waiting to lock <0x00000007b1283570> (a java.lang.Object)
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)

"qtp801808302-80-acceptor-0-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #80 prio=5 os_prio=0
tid=0x00007f7ad86df000 nid=0x7a7c runnable [0x00007f7ab64fd000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        - locked <0x00000007b1283570> (a java.lang.Object)
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)

"qtp801808302-79-selector-ServerConnectorManager@6591a1ee/23" #79 prio=5 os_prio=0 tid=0x00007f7ad1368800
nid=0x7a7b runnable [0x00007f7ab65fe000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007b13f70d0> (a sun.nio.ch.Util$2)
        - locked <0x00000007b13f70c0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007b13f6fa8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)

"qtp801808302-78-selector-ServerConnectorManager@6591a1ee/22" #78 prio=5 os_prio=0 tid=0x00000000026c2800
nid=0x7a7a runnable [0x00007f7ab66ff000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007b13f62e0> (a sun.nio.ch.Util$2)
        - locked <0x00000007b13f62d0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007b13f61b8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)

"qtp801808302-77-selector-ServerConnectorManager@6591a1ee/21" #77 prio=5 os_prio=0 tid=0x00007f7b233e5000
nid=0x7a79 runnable [0x00007f7ab6800000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007b13f54f0> (a sun.nio.ch.Util$2)
        - locked <0x00000007b13f54e0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007b13f53c8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)

"qtp801808302-76-selector-ServerConnectorManager@6591a1ee/20" #76 prio=5 os_prio=0 tid=0x00007f7ad86dd000
nid=0x7a78 runnable [0x00007f7ab6901000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007b13f4700> (a sun.nio.ch.Util$2)
        - locked <0x00000007b13f46f0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007b13f45d8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)

"qtp801808302-74-selector-ServerConnectorManager@6591a1ee/19" #74 prio=5 os_prio=0 tid=0x00007f7ad13cb800
nid=0x7a77 runnable [0x00007f7ab6a01000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007b13f3910> (a sun.nio.ch.Util$2)
        - locked <0x00000007b13f3900> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007b13f37e8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)

"qtp801808302-73-selector-ServerConnectorManager@6591a1ee/18" #73 prio=5 os_prio=0 tid=0x00000000026c1000
nid=0x7a76 runnable [0x00007f7ab6b03000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007b13f2b20> (a sun.nio.ch.Util$2)
        - locked <0x00000007b13f2b10> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007b13f29f8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)

"qtp801808302-72-selector-ServerConnectorManager@6591a1ee/17" #72 prio=5 os_prio=0 tid=0x00007f7b233e3800
nid=0x7a75 runnable [0x00007f7ab6c04000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007b13f1ce0> (a sun.nio.ch.Util$2)
        - locked <0x00000007b13f1cd0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007b13f1bb8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)

"qtp801808302-71-selector-ServerConnectorManager@6591a1ee/16" #71 prio=5 os_prio=0 tid=0x00007f7ad86db000
nid=0x7a74 runnable [0x00007f7ab6d05000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007b13e14e0> (a sun.nio.ch.Util$2)
        - locked <0x00000007b13e14d0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007b13e13b8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)

...

"Scheduler-897230287" #62 prio=5 os_prio=0 tid=0x00007f7b23302800 nid=0x7a50 waiting on condition
[0x00007f7ab770d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007bb901728> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"qtp801808302-61-selector-ServerConnectorManager@6591a1ee/7" #61 prio=5 os_prio=0 tid=0x00007f7b233b8000
nid=0x7a4f runnable [0x00007f7ab780e000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007b12fb2a0> (a sun.nio.ch.Util$2)
        - locked <0x00000007b12fb290> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007b12fb178> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)

...

"Curator-PathChildrenCache-1" #52 daemon prio=5 os_prio=0 tid=0x00007f7b2308c000 nid=0x7a47
waiting on condition [0x00007f7ab9179000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007bde07d08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"WorkManager.StatusThread" #40 daemon prio=5 os_prio=0 tid=0x00007f7b229af000 nid=0x7a45 waiting
on condition [0x00007f7ab825a000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.drill.exec.work.WorkManager$StatusThread.run(WorkManager.java:328)

"Curator-PathChildrenCache-0" #51 daemon prio=5 os_prio=0 tid=0x00007f7b2170f800 nid=0x79b2
waiting on condition [0x00007f7ab927a000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007b6b98818> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"BitServer-2" #21 daemon prio=10 os_prio=0 tid=0x00007f7b219d3800 nid=0x7981 runnable [0x00007f7ab940d000]
   java.lang.Thread.State: RUNNABLE
        at io.netty.channel.epoll.Native.epollWait0(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:148)
        at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at java.lang.Thread.run(Thread.java:745)

"UserServer-1" #41 daemon prio=10 os_prio=0 tid=0x00007f7b21846800 nid=0x797f runnable [0x00007f7abb1a6000]
   java.lang.Thread.State: RUNNABLE
        at io.netty.channel.epoll.Native.epollWait0(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:148)
        at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at java.lang.Thread.run(Thread.java:745)

"Curator-ServiceCache-0" #49 daemon prio=5 os_prio=0 tid=0x00007f7b2184d800 nid=0x796e waiting
on condition [0x00007f7abc95d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007b68066c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"Curator-Framework-0" #45 daemon prio=5 os_prio=0 tid=0x00007f7b21064800 nid=0x77ef waiting
on condition [0x00007f7aba9a6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007b6806b90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780)
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)
        at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"main-EventThread" #44 daemon prio=5 os_prio=0 tid=0x00007f7b21043000 nid=0x77ee waiting on
condition [0x00007f7abaaa7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007b6806e88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"main-SendThread(centos-01.qa.lab:5181)" #43 daemon prio=5 os_prio=0 tid=0x00007f7b20ec9000
nid=0x77ed runnable [0x00007f7abb02d000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000007b5b00220> (a sun.nio.ch.Util$2)
        - locked <0x00000007b5b00230> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007b5b001d8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1070)

"Curator-ConnectionStateManager-0" #42 daemon prio=5 os_prio=0 tid=0x00007f7b2223d000 nid=0x77ec
waiting on condition [0x00007f7ababa8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007b6807088> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
        at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
        at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
...

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f7b20cf8000 nid=0x7529 in Object.wait()
[0x00007f7ac46d5000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000007b6807f58> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f7b20cf0000 nid=0x7528 in Object.wait()
[0x00007f7ac47d6000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
        - locked <0x00000007b5b00d88> (a java.lang.ref.Reference$Lock)

...

Heap
 garbage-first heap   total 4194304K, used 293887K [0x00000006c0000000, 0x00000006c0204000,
0x00000007c0000000)
  region size 2048K, 145 young (296960K), 21 survivors (43008K)
 Metaspace       used 43409K, capacity 43751K, committed 44284K, reserved 1087488K
  class space    used 5383K, capacity 5510K, committed 5628K, reserved 1048576K

2016-02-01 11:04:08
Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):

"qtp801808302-130" #130 prio=5 os_prio=0 tid=0x00007f7ad8125000 nid=0xc5b waiting on condition
[0x00007f7ab61fa000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007b69100a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:513)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:569)
        at java.lang.Thread.run(Thread.java:745)

"DestroyJavaVM" #84 prio=5 os_prio=0 tid=0x00007f7b20012000 nid=0x74c0 waiting on condition
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"qtp801808302-81-acceptor-1-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #81 prio=5 os_prio=0
tid=0x00007f7b233e7000 nid=0x7a7d waiting for monitor entry [0x00007f7ab63fc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
        - waiting to lock <0x00000007b1283570> (a java.lang.Object)
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message