drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sudheesh Katkam (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (DRILL-4330) Long running SQL query hangs once Foreman node is killed
Date Thu, 07 Apr 2016 17:31:25 GMT

     [ https://issues.apache.org/jira/browse/DRILL-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sudheesh Katkam resolved DRILL-4330.
------------------------------------
    Resolution: Fixed

Fixed through DRILL-3743.

> Long running SQL query hangs once Foreman node is killed
> --------------------------------------------------------
>
>                 Key: DRILL-4330
>                 URL: https://issues.apache.org/jira/browse/DRILL-4330
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Flow
>    Affects Versions: 1.4.0
>         Environment: 4 node cluster CentOS
>            Reporter: Khurram Faraaz
>            Assignee: Sudheesh Katkam
>         Attachments: drillbit.out
>
>
> Summary : Once Foreman node Drillbit is killed, long running query just hangs and no
profile information is written to Web UI. That long running query was issued from the Foreman
node.
> MapR Drill 1.4.0 GA
> MapR FS 5.0.0 GA
> JDK8
> 4 node CentOS cluster
> ./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
> Issue a long running select query over JSON data
> Immediately kill the Drillbit on Foreman node (ps -eaf | grep Drillbit), kill -9 PID
> The long running query hangs on sqlline prompt, there are no messages/errors/Exceptions
reported on sqlline prompt.
> On the Web UI there is no profile reported for the long running query that was running
on the Drillbit that was killed.
> Question (1) : Why was there no profile reported/written on the Web UI for that long
running query ? In a real production scenario user will not know what query was under execution
at the point when Foreman went down. 
> Question (2) : Why does the long running query not terminate, once the foreman was killed
? from the drillbit.log snippet we do not see any CANCELED/TERMINATED message for that query,
why ?
> Snippet from drillbit.log on the foreman node. 
> {noformat}
> 2016-02-01 10:59:20,917 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.drill.exec.work.foreman.Foreman
- Query text for query id 2950c576-b2d2-5bc3-e9b5-ff4414d088c0: select * from `twoKeyJsn.json`
> 2016-02-01 10:59:21,067 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.create() took 1 ms
> 2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,155 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatuses() took 0 ms, numFiles: 1
> 2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.e.s.schedule.BlockMapBuilder
- Get block maps: Executed 1 out of 1 using 1 threads. Time: 90ms total, 90.891938ms avg,
90ms max.
> 2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.e.s.schedule.BlockMapBuilder
- Get block maps: Executed 1 out of 1 using 1 threads. Earliest start: 18.280000 μs, Latest
start: 18.280000 μs, Average start: 18.280000 μs .
> 2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State change requested AWAITING_ALLOCATION -->
RUNNING
> 2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO  o.a.d.e.w.f.FragmentStatusReporter
- 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State to report: RUNNING
> {noformat}
> Doing kill -3 PID on the non foreman node for the Drillbit process gives us stack trace
in drillbit.out
> {noformat}
> 2016-02-01 11:03:31
> Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):
> "qtp801808302-129" #129 prio=5 os_prio=0 tid=0x00007f7ad8127000 nid=0xaad waiting on
condition [0x00007f7ab62fb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007b69100a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:513)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:569)
>         at java.lang.Thread.run(Thread.java:745)
> "DestroyJavaVM" #84 prio=5 os_prio=0 tid=0x00007f7b20012000 nid=0x74c0 waiting on condition
[0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "qtp801808302-81-acceptor-1-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #81 prio=5
os_prio=0 tid=0x00007f7b233e7000 nid=0x7a7d waiting for monitor entry [0x00007f7ab63fc000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
>         - waiting to lock <0x00000007b1283570> (a java.lang.Object)
>         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
>         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-80-acceptor-0-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #80 prio=5
os_prio=0 tid=0x00007f7ad86df000 nid=0x7a7c runnable [0x00007f7ab64fd000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>         - locked <0x00000007b1283570> (a java.lang.Object)
>         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
>         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-79-selector-ServerConnectorManager@6591a1ee/23" #79 prio=5 os_prio=0 tid=0x00007f7ad1368800
nid=0x7a7b runnable [0x00007f7ab65fe000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000007b13f70d0> (a sun.nio.ch.Util$2)
>         - locked <0x00000007b13f70c0> (a java.util.Collections$UnmodifiableSet)
>         - locked <0x00000007b13f6fa8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
>         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-78-selector-ServerConnectorManager@6591a1ee/22" #78 prio=5 os_prio=0 tid=0x00000000026c2800
nid=0x7a7a runnable [0x00007f7ab66ff000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000007b13f62e0> (a sun.nio.ch.Util$2)
>         - locked <0x00000007b13f62d0> (a java.util.Collections$UnmodifiableSet)
>         - locked <0x00000007b13f61b8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
>         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-77-selector-ServerConnectorManager@6591a1ee/21" #77 prio=5 os_prio=0 tid=0x00007f7b233e5000
nid=0x7a79 runnable [0x00007f7ab6800000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000007b13f54f0> (a sun.nio.ch.Util$2)
>         - locked <0x00000007b13f54e0> (a java.util.Collections$UnmodifiableSet)
>         - locked <0x00000007b13f53c8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
>         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-76-selector-ServerConnectorManager@6591a1ee/20" #76 prio=5 os_prio=0 tid=0x00007f7ad86dd000
nid=0x7a78 runnable [0x00007f7ab6901000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000007b13f4700> (a sun.nio.ch.Util$2)
>         - locked <0x00000007b13f46f0> (a java.util.Collections$UnmodifiableSet)
>         - locked <0x00000007b13f45d8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
>         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-74-selector-ServerConnectorManager@6591a1ee/19" #74 prio=5 os_prio=0 tid=0x00007f7ad13cb800
nid=0x7a77 runnable [0x00007f7ab6a01000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000007b13f3910> (a sun.nio.ch.Util$2)
>         - locked <0x00000007b13f3900> (a java.util.Collections$UnmodifiableSet)
>         - locked <0x00000007b13f37e8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
>         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-73-selector-ServerConnectorManager@6591a1ee/18" #73 prio=5 os_prio=0 tid=0x00000000026c1000
nid=0x7a76 runnable [0x00007f7ab6b03000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000007b13f2b20> (a sun.nio.ch.Util$2)
>         - locked <0x00000007b13f2b10> (a java.util.Collections$UnmodifiableSet)
>         - locked <0x00000007b13f29f8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
>         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-72-selector-ServerConnectorManager@6591a1ee/17" #72 prio=5 os_prio=0 tid=0x00007f7b233e3800
nid=0x7a75 runnable [0x00007f7ab6c04000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000007b13f1ce0> (a sun.nio.ch.Util$2)
>         - locked <0x00000007b13f1cd0> (a java.util.Collections$UnmodifiableSet)
>         - locked <0x00000007b13f1bb8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
>         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-71-selector-ServerConnectorManager@6591a1ee/16" #71 prio=5 os_prio=0 tid=0x00007f7ad86db000
nid=0x7a74 runnable [0x00007f7ab6d05000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000007b13e14e0> (a sun.nio.ch.Util$2)
>         - locked <0x00000007b13e14d0> (a java.util.Collections$UnmodifiableSet)
>         - locked <0x00000007b13e13b8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
>         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:745)
> ...
> "Scheduler-897230287" #62 prio=5 os_prio=0 tid=0x00007f7b23302800 nid=0x7a50 waiting
on condition [0x00007f7ab770d000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007bb901728> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> "qtp801808302-61-selector-ServerConnectorManager@6591a1ee/7" #61 prio=5 os_prio=0 tid=0x00007f7b233b8000
nid=0x7a4f runnable [0x00007f7ab780e000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000007b12fb2a0> (a sun.nio.ch.Util$2)
>         - locked <0x00000007b12fb290> (a java.util.Collections$UnmodifiableSet)
>         - locked <0x00000007b12fb178> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
>         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
>         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:745)
> ...
> "Curator-PathChildrenCache-1" #52 daemon prio=5 os_prio=0 tid=0x00007f7b2308c000 nid=0x7a47
waiting on condition [0x00007f7ab9179000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007bde07d08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> "WorkManager.StatusThread" #40 daemon prio=5 os_prio=0 tid=0x00007f7b229af000 nid=0x7a45
waiting on condition [0x00007f7ab825a000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at org.apache.drill.exec.work.WorkManager$StatusThread.run(WorkManager.java:328)
> "Curator-PathChildrenCache-0" #51 daemon prio=5 os_prio=0 tid=0x00007f7b2170f800 nid=0x79b2
waiting on condition [0x00007f7ab927a000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007b6b98818> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> "BitServer-2" #21 daemon prio=10 os_prio=0 tid=0x00007f7b219d3800 nid=0x7981 runnable
[0x00007f7ab940d000]
>    java.lang.Thread.State: RUNNABLE
>         at io.netty.channel.epoll.Native.epollWait0(Native Method)
>         at io.netty.channel.epoll.Native.epollWait(Native.java:148)
>         at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
>         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
> "UserServer-1" #41 daemon prio=10 os_prio=0 tid=0x00007f7b21846800 nid=0x797f runnable
[0x00007f7abb1a6000]
>    java.lang.Thread.State: RUNNABLE
>         at io.netty.channel.epoll.Native.epollWait0(Native Method)
>         at io.netty.channel.epoll.Native.epollWait(Native.java:148)
>         at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
>         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>         at java.lang.Thread.run(Thread.java:745)
> "Curator-ServiceCache-0" #49 daemon prio=5 os_prio=0 tid=0x00007f7b2184d800 nid=0x796e
waiting on condition [0x00007f7abc95d000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007b68066c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> "Curator-Framework-0" #45 daemon prio=5 os_prio=0 tid=0x00007f7b21064800 nid=0x77ef waiting
on condition [0x00007f7aba9a6000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007b6806b90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
>         at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
>         at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780)
>         at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)
>         at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> "main-EventThread" #44 daemon prio=5 os_prio=0 tid=0x00007f7b21043000 nid=0x77ee waiting
on condition [0x00007f7abaaa7000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007b6806e88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
> "main-SendThread(centos-01.qa.lab:5181)" #43 daemon prio=5 os_prio=0 tid=0x00007f7b20ec9000
nid=0x77ed runnable [0x00007f7abb02d000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
>         - locked <0x00000007b5b00220> (a sun.nio.ch.Util$2)
>         - locked <0x00000007b5b00230> (a java.util.Collections$UnmodifiableSet)
>         - locked <0x00000007b5b001d8> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
>         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
>         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1070)
> "Curator-ConnectionStateManager-0" #42 daemon prio=5 os_prio=0 tid=0x00007f7b2223d000
nid=0x77ec waiting on condition [0x00007f7ababa8000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007b6807088> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>         at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
>         at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
>         at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
>         at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> ...
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f7b20cf8000 nid=0x7529 in Object.wait()
[0x00007f7ac46d5000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
>         - locked <0x00000007b6807f58> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
>         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f7b20cf0000 nid=0x7528 in
Object.wait() [0x00007f7ac47d6000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         at java.lang.Object.wait(Object.java:502)
>         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
>         - locked <0x00000007b5b00d88> (a java.lang.ref.Reference$Lock)
> ...
> Heap
>  garbage-first heap   total 4194304K, used 293887K [0x00000006c0000000, 0x00000006c0204000,
0x00000007c0000000)
>   region size 2048K, 145 young (296960K), 21 survivors (43008K)
>  Metaspace       used 43409K, capacity 43751K, committed 44284K, reserved 1087488K
>   class space    used 5383K, capacity 5510K, committed 5628K, reserved 1048576K
> 2016-02-01 11:04:08
> Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):
> "qtp801808302-130" #130 prio=5 os_prio=0 tid=0x00007f7ad8125000 nid=0xc5b waiting on
condition [0x00007f7ab61fa000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00000007b69100a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:513)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:569)
>         at java.lang.Thread.run(Thread.java:745)
> "DestroyJavaVM" #84 prio=5 os_prio=0 tid=0x00007f7b20012000 nid=0x74c0 waiting on condition
[0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "qtp801808302-81-acceptor-1-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #81 prio=5
os_prio=0 tid=0x00007f7b233e7000 nid=0x7a7d waiting for monitor entry [0x00007f7ab63fc000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
>         - waiting to lock <0x00000007b1283570> (a java.lang.Object)
>         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
>         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>         at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message