drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Khurram Faraaz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-4338) Concurrent query remains in CANCELLATION_REQUESTED state
Date Wed, 03 Feb 2016 07:10:39 GMT

    [ https://issues.apache.org/jira/browse/DRILL-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129934#comment-15129934
] 

Khurram Faraaz commented on DRILL-4338:
---------------------------------------

Likewise, while the concurrent set of queries are under execution, if the Drillbit on Foreman
node is killed (kill -9 PID), the java program that was under execution hangs without terminating.
Here is the stack for the hung java process.

{noformat}
 [root@centos-01 logs]# ps -eaf | grep Concurren
root     17772  1827  0 06:04 pts/9    00:00:00 grep Concurren
root     31081  2411  2 05:50 pts/7    00:00:18 java ConcurrencyTest
[root@centos-01 logs]# jstack 31081
2016-02-03 06:05:26
Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):

"Attach Listener" #35 daemon prio=9 os_prio=0 tid=0x00007ff3f0001000 nid=0x488b waiting on
condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"DestroyJavaVM" #34 prio=5 os_prio=0 tid=0x00007ff4a8011000 nid=0x796a waiting on condition
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"pool-1-thread-12" #33 prio=5 os_prio=0 tid=0x00007ff4a8e54000 nid=0x79c3 waiting on condition
[0x00007ff4765f3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d0718178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl$ResultsListener.getNext(DrillResultSetImpl.java:2097)
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:175)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at ConcurrencyTest.executeQuery(ConcurrencyTest.java:55)
	at ConcurrencyTest.SELECTData(ConcurrencyTest.java:37)
	at ConcurrencyTest.run(ConcurrencyTest.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-11" #32 prio=5 os_prio=0 tid=0x00007ff4a8e52800 nid=0x79c2 waiting on condition
[0x00007ff4766f4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d0720178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl$ResultsListener.getNext(DrillResultSetImpl.java:2097)
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:175)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at ConcurrencyTest.executeQuery(ConcurrencyTest.java:55)
	at ConcurrencyTest.SELECTData(ConcurrencyTest.java:37)
	at ConcurrencyTest.run(ConcurrencyTest.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-10" #31 prio=5 os_prio=0 tid=0x00007ff4a8e51000 nid=0x79c1 waiting on condition
[0x00007ff4767f5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d0728178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl$ResultsListener.getNext(DrillResultSetImpl.java:2097)
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:175)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at ConcurrencyTest.executeQuery(ConcurrencyTest.java:55)
	at ConcurrencyTest.SELECTData(ConcurrencyTest.java:38)
	at ConcurrencyTest.run(ConcurrencyTest.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-9" #30 prio=5 os_prio=0 tid=0x00007ff4a8e4f000 nid=0x79c0 waiting on condition
[0x00007ff4768f6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d0730178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl$ResultsListener.getNext(DrillResultSetImpl.java:2097)
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:175)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at ConcurrencyTest.executeQuery(ConcurrencyTest.java:55)
	at ConcurrencyTest.SELECTData(ConcurrencyTest.java:38)
	at ConcurrencyTest.run(ConcurrencyTest.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-8" #29 prio=5 os_prio=0 tid=0x00007ff4a8e4d800 nid=0x79bf waiting on condition
[0x00007ff4769f6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d0738178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl$ResultsListener.getNext(DrillResultSetImpl.java:2097)
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:175)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at ConcurrencyTest.executeQuery(ConcurrencyTest.java:55)
	at ConcurrencyTest.SELECTData(ConcurrencyTest.java:37)
	at ConcurrencyTest.run(ConcurrencyTest.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-7" #28 prio=5 os_prio=0 tid=0x00007ff4a8e4b800 nid=0x79be waiting on condition
[0x00007ff476af8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d0740178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl$ResultsListener.getNext(DrillResultSetImpl.java:2097)
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:175)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at ConcurrencyTest.executeQuery(ConcurrencyTest.java:55)
	at ConcurrencyTest.SELECTData(ConcurrencyTest.java:38)
	at ConcurrencyTest.run(ConcurrencyTest.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-6" #27 prio=5 os_prio=0 tid=0x00007ff4a8e4a000 nid=0x79bd waiting on condition
[0x00007ff476bf9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d0748178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl$ResultsListener.getNext(DrillResultSetImpl.java:2097)
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:175)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at ConcurrencyTest.executeQuery(ConcurrencyTest.java:55)
	at ConcurrencyTest.SELECTData(ConcurrencyTest.java:38)
	at ConcurrencyTest.run(ConcurrencyTest.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-5" #26 prio=5 os_prio=0 tid=0x00007ff4a8e48800 nid=0x79bc waiting on condition
[0x00007ff476cfa000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d0710300> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl$ResultsListener.getNext(DrillResultSetImpl.java:2097)
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:175)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at ConcurrencyTest.executeQuery(ConcurrencyTest.java:55)
	at ConcurrencyTest.SELECTData(ConcurrencyTest.java:38)
	at ConcurrencyTest.run(ConcurrencyTest.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-4" #25 prio=5 os_prio=0 tid=0x00007ff4a8e46800 nid=0x79bb waiting on condition
[0x00007ff476dfb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d0750178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl$ResultsListener.getNext(DrillResultSetImpl.java:2097)
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:175)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at ConcurrencyTest.executeQuery(ConcurrencyTest.java:55)
	at ConcurrencyTest.SELECTData(ConcurrencyTest.java:37)
	at ConcurrencyTest.run(ConcurrencyTest.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-3" #24 prio=5 os_prio=0 tid=0x00007ff4a8e45000 nid=0x79ba waiting on condition
[0x00007ff476efc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d0700720> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl$ResultsListener.getNext(DrillResultSetImpl.java:2097)
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:175)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at ConcurrencyTest.executeQuery(ConcurrencyTest.java:55)
	at ConcurrencyTest.SELECTData(ConcurrencyTest.java:38)
	at ConcurrencyTest.run(ConcurrencyTest.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-2" #23 prio=5 os_prio=0 tid=0x00007ff4a8e44000 nid=0x79b9 runnable [0x00007ff476ffd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d0758178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl$ResultsListener.getNext(DrillResultSetImpl.java:2097)
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:175)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at ConcurrencyTest.executeQuery(ConcurrencyTest.java:55)
	at ConcurrencyTest.SELECTData(ConcurrencyTest.java:38)
	at ConcurrencyTest.run(ConcurrencyTest.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-1" #22 prio=5 os_prio=0 tid=0x00007ff4a8e41800 nid=0x79b8 waiting on condition
[0x00007ff47c13f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d0760178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl$ResultsListener.getNext(DrillResultSetImpl.java:2097)
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:175)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at ConcurrencyTest.executeQuery(ConcurrencyTest.java:55)
	at ConcurrencyTest.SELECTData(ConcurrencyTest.java:38)
	at ConcurrencyTest.run(ConcurrencyTest.java:23)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"threadDeathWatcher-2-1" #21 daemon prio=1 os_prio=0 tid=0x00007ff3ac017800 nid=0x79b7 waiting
on condition [0x00007ff47c241000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)

"Client-1" #19 daemon prio=10 os_prio=0 tid=0x00007ff4a8e40000 nid=0x79b6 runnable [0x00007ff47c542000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000006d07702b0> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000006d07702a0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000006d07701c0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)

"Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007ff4a810d800 nid=0x79b3 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007ff4a810a800 nid=0x79b2 waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007ff4a8108800 nid=0x79b1 waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007ff4a8106800 nid=0x79ad waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007ff4a8104800 nid=0x7994 waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007ff4a8102800 nid=0x7993 waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007ff4a8100800 nid=0x7992 waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007ff4a80fe800 nid=0x7991 waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007ff4a80f4800 nid=0x7990 waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007ff4a80f2000 nid=0x798f waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007ff4a80f0000 nid=0x798e waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007ff4a80ee800 nid=0x798d waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007ff4a80e1000 nid=0x798c waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007ff4a80df000 nid=0x798b runnable
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007ff4a80b0000 nid=0x798a in Object.wait()
[0x00007ff47edec000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000006d0768990> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
	- locked <0x00000006d0768990> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007ff4a80ae000 nid=0x7989 in Object.wait()
[0x00007ff47eeed000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000006d07805e0> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
	- locked <0x00000006d07805e0> (a java.lang.ref.Reference$Lock)

"VM Thread" os_prio=0 tid=0x00007ff4a80a9000 nid=0x7988 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007ff4a8026800 nid=0x796b runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007ff4a8028000 nid=0x796c runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007ff4a802a000 nid=0x796d runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007ff4a802b800 nid=0x796e runnable

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007ff4a802d800 nid=0x796f runnable

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007ff4a802f000 nid=0x7970 runnable

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007ff4a8031000 nid=0x7971 runnable

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007ff4a8032800 nid=0x7972 runnable

"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007ff4a8034800 nid=0x7973 runnable

"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007ff4a8036000 nid=0x7974 runnable

"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007ff4a8038000 nid=0x7975 runnable

"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007ff4a8039800 nid=0x7976 runnable

"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007ff4a803b800 nid=0x7977 runnable

"GC task thread#13 (ParallelGC)" os_prio=0 tid=0x00007ff4a803d000 nid=0x7978 runnable

"GC task thread#14 (ParallelGC)" os_prio=0 tid=0x00007ff4a803f000 nid=0x7979 runnable

"GC task thread#15 (ParallelGC)" os_prio=0 tid=0x00007ff4a8040800 nid=0x7985 runnable

"GC task thread#16 (ParallelGC)" os_prio=0 tid=0x00007ff4a8042800 nid=0x7986 runnable

"GC task thread#17 (ParallelGC)" os_prio=0 tid=0x00007ff4a8044000 nid=0x7987 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007ff4a8110000 nid=0x79b4 waiting on condition

JNI global references: 256
 {noformat}

> Concurrent query remains in CANCELLATION_REQUESTED state 
> ---------------------------------------------------------
>
>                 Key: DRILL-4338
>                 URL: https://issues.apache.org/jira/browse/DRILL-4338
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Flow
>    Affects Versions: 1.4.0
>         Environment: 4 node cluster CentOS
>            Reporter: Khurram Faraaz
>         Attachments: ConcurrencyTest.java, query_In_cancellation_requested_state.png
>
>
> Execute a query concurrently through a Java program and while the java program is under
execution (executing SQL queries concurrently) issue Ctrl-C on the prompt where the java program
was being executed.
> Here are two observations, 
> (1) There is an Exception in drillbit.log.
> (2) Once Ctrl-C was issued to the java program, queries that were under execution at
that point of time, move from FAILED state to CANCELLATION_REQUESTED state, they do not end
up in CANCELED state. Ideally that last state of these queries should be CANCELED state and
not CANCELLATION_REQUESTED. 
> Snippet from drillbit.log
> {noformat}
> 2016-02-02 06:21:21,903 [294fb51d-8a4c-c099-dc90-97434056e3d7:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: State change requested AWAITING_ALLOCATION -->
RUNNING
> 2016-02-02 06:21:21,903 [294fb51d-8a4c-c099-dc90-97434056e3d7:frag:0:0] INFO  o.a.d.e.w.f.FragmentStatusReporter
- 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: State to report: RUNNING
> 2016-02-02 06:21:48,560 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception
in RPC communication.  Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087 (user
client).  Closing connection.
> java.io.IOException: syscall:read(...)() failed: Connection reset by peer
> 2016-02-02 06:21:48,562 [UserServer-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0:
State change requested RUNNING --> FAILED
> 2016-02-02 06:21:48,562 [UserServer-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-f424-6adc-d668-1659e4353698:0:0:
State change requested RUNNING --> FAILED
> 2016-02-02 06:21:48,562 [UserServer-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0:
State change requested RUNNING --> FAILED
> 2016-02-02 06:21:48,563 [UserServer-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 294fb51e-5de9-0919-be56-52f75a0532f1:0:0:
State change requested RUNNING --> FAILED
> 2016-02-02 06:21:48,573 [CONTROL-rpc-event-queue] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 294fb51d-f424-6adc-d668-1659e4353698:0:0: State change requested FAILED --> CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,573 [CONTROL-rpc-event-queue] WARN  o.a.d.e.w.fragment.FragmentExecutor
- 294fb51d-f424-6adc-d668-1659e4353698:0:0: Ignoring unexpected state transition FAILED -->
CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,580 [CONTROL-rpc-event-queue] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 294fb51e-5de9-0919-be56-52f75a0532f1:0:0: State change requested FAILED --> CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,580 [CONTROL-rpc-event-queue] WARN  o.a.d.e.w.fragment.FragmentExecutor
- 294fb51e-5de9-0919-be56-52f75a0532f1:0:0: Ignoring unexpected state transition FAILED -->
CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,588 [CONTROL-rpc-event-queue] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0: State change requested FAILED --> CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,588 [CONTROL-rpc-event-queue] WARN  o.a.d.e.w.fragment.FragmentExecutor
- 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0: Ignoring unexpected state transition FAILED -->
CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,596 [CONTROL-rpc-event-queue] INFO  o.a.d.e.w.fragment.FragmentExecutor
- 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: State change requested FAILED --> CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,596 [CONTROL-rpc-event-queue] WARN  o.a.d.e.w.fragment.FragmentExecutor
- 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: Ignoring unexpected state transition FAILED -->
CANCELLATION_REQUESTED
> 2016-02-02 06:21:48,597 [UserServer-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-f424-6adc-d668-1659e4353698:0:0:
State change requested FAILED --> FAILED
> 2016-02-02 06:21:48,599 [UserServer-1] WARN  o.a.d.exec.rpc.RpcExceptionHandler - Exception
occurred with closed channel.  Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087
(user client)
> io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message.
> at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98)
~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> 2016-02-02 06:21:48,600 [UserServer-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 294fb51e-5de9-0919-be56-52f75a0532f1:0:0:
State change requested FAILED --> FAILED
> 2016-02-02 06:21:48,600 [UserServer-1] WARN  o.a.d.exec.rpc.RpcExceptionHandler - Exception
occurred with closed channel.  Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087
(user client)
> io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message.
>         at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98)
~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> 2016-02-02 06:21:48,601 [UserServer-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0:
State change requested FAILED --> FAILED
> 2016-02-02 06:21:48,601 [UserServer-1] WARN  o.a.d.exec.rpc.RpcExceptionHandler - Exception
occurred with closed channel.  Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087
(user client)
> io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message.
>         at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98)
~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> 2016-02-02 06:21:48,601 [UserServer-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0:
State change requested FAILED --> FAILED
> 2016-02-02 06:21:48,601 [UserServer-1] WARN  o.a.d.exec.rpc.RpcExceptionHandler - Exception
occurred with closed channel.  Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087
(user client)
> io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message.
>         at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98)
~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> 2016-02-02 06:21:48,602 [UserServer-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-f424-6adc-d668-1659e4353698:0:0:
State change requested FAILED --> FAILED
> 2016-02-02 06:21:48,602 [UserServer-1] WARN  o.a.d.exec.rpc.RpcExceptionHandler - Exception
occurred with closed channel.  Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087
(user client)
> io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message.
>         at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98)
~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> 2016-02-02 06:21:48,602 [UserServer-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 294fb51e-5de9-0919-be56-52f75a0532f1:0:0:
State change requested FAILED --> FAILED
> 2016-02-02 06:21:48,603 [UserServer-1] WARN  o.a.d.exec.rpc.RpcExceptionHandler - Exception
occurred with closed channel.  Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087
(user client)
> io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message.
>         at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98)
~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> 2016-02-02 06:21:48,603 [UserServer-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0:
State change requested FAILED --> FAILED
> 2016-02-02 06:21:48,603 [UserServer-1] WARN  o.a.d.exec.rpc.RpcExceptionHandler - Exception
occurred with closed channel.  Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087
(user client)
> io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message.
>         at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98)
~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> 2016-02-02 06:21:48,603 [UserServer-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0:
State change requested FAILED --> FAILED
> 2016-02-02 06:21:48,604 [UserServer-1] WARN  o.a.d.exec.rpc.RpcExceptionHandler - Exception
occurred with closed channel.  Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087
(user client)
> io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message.
>         at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98)
~[netty-codec-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message