drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Khurram Faraaz (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-5561) IllegalArgumentException: Self-suppression not permitted, seen when foreman Drillbit is killed
Date Sat, 03 Jun 2017 07:39:04 GMT
Khurram Faraaz created DRILL-5561:
-------------------------------------

             Summary: IllegalArgumentException: Self-suppression not permitted, seen when
foreman Drillbit is killed
                 Key: DRILL-5561
                 URL: https://issues.apache.org/jira/browse/DRILL-5561
             Project: Apache Drill
          Issue Type: Bug
          Components: Execution - Flow
    Affects Versions: 1.11.0
         Environment: 3 node Centos cluster
            Reporter: Khurram Faraaz
            Priority: Minor



Drill 1.11.0 running on three nodes

kill foreman Drillbit, [root@centos-01 ~]# kill -9 15852; while a long running SQL query (TPCDS
Query 11) is under execution

drillbit.log from centos-01.qa.lab (foreman Drillbit) has no Exceptions or Errors logged in
it.

snippet of drillbit.log from centos-02.qa.lab ( non foreman Drillbit )

{noformat}
2017-05-30 08:21:32,834 [BitClient-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 26d2d481-d949-1de8-9497-c1492f086501:12:0:
State change requested RUNNING --> FAILED
2017-05-30 08:21:32,834 [BitServer-9] WARN  o.a.d.exec.rpc.RpcExceptionHandler - Exception
occurred with closed channel.  Connection: /10.10.100.202:31012 <--> /10.10.100.201:50773
(data server)
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_65]
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_65]
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_65]
        at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_65]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_65]
        at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311)
~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.buffer.WrappedByteBuf.setBytes(WrappedByteBuf.java:407) ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.buffer.UnsafeDirectLittleEndian.setBytes(UnsafeDirectLittleEndian.java:30)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
        at io.netty.buffer.DrillBuf.setBytes(DrillBuf.java:770) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
        at io.netty.buffer.MutableWrappedByteBuf.setBytes(MutableWrappedByteBuf.java:280)
~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
        at io.netty.buffer.ExpandableByteBuf.setBytes(ExpandableByteBuf.java:26) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:4.0.27.Final]
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881) ~[netty-buffer-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
2017-05-30 08:21:32,848 [BitClient-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 26d2d481-d949-1de8-9497-c1492f086501:16:0:
State change requested RUNNING --> FAILED
2017-05-30 08:21:32,850 [BitClient-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 26d2d481-d949-1de8-9497-c1492f086501:14:0:
State change requested RUNNING --> FAILED
2017-05-30 08:21:32,851 [BitClient-1] WARN  o.apache.drill.exec.rpc.RequestIdMap - Failure
while attempting to fail rpc response.
java.lang.IllegalArgumentException: Self-suppression not permitted
        at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[na:1.8.0_65]
        at org.apache.drill.common.DeferredException.addException(DeferredException.java:88)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:409)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor.access$700(FragmentExecutor.java:55)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor$ExecutorStateImpl.fail(FragmentExecutor.java:423)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.ops.FragmentContext.fail(FragmentContext.java:214) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.ops.FragmentContext$1.accept(FragmentContext.java:96) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.ops.FragmentContext$1.accept(FragmentContext.java:93) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.ops.StatusHandler.failed(StatusHandler.java:42) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.ListeningCommand$DeferredRpcOutcome.failed(ListeningCommand.java:50)
~[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.data.DataTunnel$ThrottlingOutcomeListener.failed(DataTunnel.java:127)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:134)
~[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:74)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:64)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692) [hppc-0.7.1.jar:na]
        at org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:58) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:192)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.AbstractClientConnection.channelClosed(AbstractClientConnection.java:97)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:167)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:205)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:192)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406) [netty-common-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: org.apache.drill.exec.rpc.ChannelClosedException: Channel closed /10.10.100.202:46969
<--> centos-01.qa.lab/10.10.100.201:31012.
        at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:164)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        ... 18 common frames omitted
2017-05-30 08:21:32,852 [BitClient-1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 26d2d481-d949-1de8-9497-c1492f086501:10:0:
State change requested RUNNING --> FAILED
2017-05-30 08:21:32,853 [BitClient-1] WARN  o.apache.drill.exec.rpc.RequestIdMap - Failure
while attempting to fail rpc response.
java.lang.IllegalArgumentException: Self-suppression not permitted
        at java.lang.Throwable.addSuppressed(Throwable.java:1043) ~[na:1.8.0_65]
        at org.apache.drill.common.DeferredException.addException(DeferredException.java:88)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97)
~[drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:409)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor.access$700(FragmentExecutor.java:55)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor$ExecutorStateImpl.fail(FragmentExecutor.java:423)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.ops.FragmentContext.fail(FragmentContext.java:214) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.ops.FragmentContext$1.accept(FragmentContext.java:96) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.ops.FragmentContext$1.accept(FragmentContext.java:93) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.ops.StatusHandler.failed(StatusHandler.java:42) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.ListeningCommand$DeferredRpcOutcome.failed(ListeningCommand.java:50)
~[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.data.DataTunnel$ThrottlingOutcomeListener.failed(DataTunnel.java:127)
~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:134)
~[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:74)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:64)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692) [hppc-0.7.1.jar:na]
        at org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:58) [drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:192)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.AbstractClientConnection.channelClosed(AbstractClientConnection.java:97)
[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:167)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:205)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:192)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406) [netty-common-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
...
...
2017-05-30 08:22:14,210 [BitServer-3] ERROR o.apache.drill.exec.rpc.BasicClient - Failed to
establish connection
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: centos-01.qa.lab/10.10.100.201:31011
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:47) ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:225)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:212)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424) [netty-common-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:268)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:284)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: java.net.ConnectException: Connection refused: centos-01.qa.lab/10.10.100.201:31011
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_65]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_65]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:281)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        ... 6 common frames omitted
{noformat}

snippet of drillbit.log from centos-04.qa.lab (another non foreman Drillbit)

{noformat}
2017-05-30 08:21:19,892 [26d2d481-d949-1de8-9497-c1492f086501:frag:8:2] INFO  o.a.d.e.w.f.FragmentStatusReporter
- 26d2d481-d949-1de8-9497-c1492f086501:8:2: State to report: RUNNING
2017-05-30 08:21:35,723 [BitServer-10] ERROR o.apache.drill.exec.rpc.BasicClient - Failed
to establish connection
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: centos-01.qa.lab/10.10.100.201:31011
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:47) ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:225)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionHandler.operationComplete(BasicClient.java:212)
[drill-rpc-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424) [netty-common-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:268)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:284)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
[netty-common-4.0.27.Final.jar:4.0.27.Final]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: java.net.ConnectException: Connection refused: centos-01.qa.lab/10.10.100.201:31011
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_65]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_65]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:281)
[netty-transport-4.0.27.Final.jar:4.0.27.Final]
        ... 6 common frames omitted
{noformat}

Adding details from [~paul-rogers] email about this problem
{noformat}
The message appears to come from Netty, it is not a Drill string.

I think what this is saying is that we did something wrong in Netty and Netty is reporting
a problem that we’ve violated its API rules.

We should figure out what we did wrong and fix it so that we don’t use Netty wrong.

Since we killed a Drillbit, the message should be “Connection to a.b.c.d dropped” —
something the user can understand.
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message