Return-Path: X-Original-To: apmail-drill-issues-archive@minotaur.apache.org Delivered-To: apmail-drill-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AB5FE19832 for ; Sun, 3 Apr 2016 20:14:25 +0000 (UTC) Received: (qmail 93556 invoked by uid 500); 3 Apr 2016 20:14:25 -0000 Delivered-To: apmail-drill-issues-archive@drill.apache.org Received: (qmail 93527 invoked by uid 500); 3 Apr 2016 20:14:25 -0000 Mailing-List: contact issues-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list issues@drill.apache.org Received: (qmail 93513 invoked by uid 99); 3 Apr 2016 20:14:25 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Apr 2016 20:14:25 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 732CC2C14F6 for ; Sun, 3 Apr 2016 20:14:25 +0000 (UTC) Date: Sun, 3 Apr 2016 20:14:25 +0000 (UTC) From: "Sudheesh Katkam (JIRA)" To: issues@drill.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (DRILL-4338) Concurrent query remains in CANCELLATION_REQUESTED state MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/DRILL-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15223494#comment-15223494 ] Sudheesh Katkam commented on DRILL-4338: ---------------------------------------- Issuing control + C is cancellation only though sqlline. To issue a cancel through JDBC, use the [Statement#cancel|https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#cancel()] method; I think sqlline maps control + C to the cancel method. So issuing a control + C in your application is causing the connection to be [closed somewhat nicely|http://stackoverflow.com/questions/1434451/what-does-connection-reset-by-peer-mean]. Regarding the two observations: (1) Drill logs the exception to note that the failure is due to connection drop; the admin might be interested in this 'unusual' event. (2) The query might end up in CANCELED or FAILED state. If Drill was sending data to the client when it died, the query is failed because the client does not acknowledge receiving data. Otherwise the query is cancelled. There are two other things to investigate: (1) The queries remain in CANCELLATION_REQUESTED state. Can you gather the jstack output of the drillbit processes? This will help us track why the cancellation or failure is not fully processed. (2) If the bit that is the Foreman is killed, the queries hang. This is same issue as DRILL-3743. > Concurrent query remains in CANCELLATION_REQUESTED state > --------------------------------------------------------- > > Key: DRILL-4338 > URL: https://issues.apache.org/jira/browse/DRILL-4338 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow > Affects Versions: 1.4.0 > Environment: 4 node cluster CentOS > Reporter: Khurram Faraaz > Attachments: ConcurrencyTest.java, query_In_cancellation_requested_state.png > > > Execute a query concurrently through a Java program and while the java program is under execution (executing SQL queries concurrently) issue Ctrl-C on the prompt where the java program was being executed. > Here are two observations, > (1) There is an Exception in drillbit.log. > (2) Once Ctrl-C was issued to the java program, queries that were under execution at that point of time, move from FAILED state to CANCELLATION_REQUESTED state, they do not end up in CANCELED state. Ideally that last state of these queries should be CANCELED state and not CANCELLATION_REQUESTED. > Snippet from drillbit.log > {noformat} > 2016-02-02 06:21:21,903 [294fb51d-8a4c-c099-dc90-97434056e3d7:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: State change requested AWAITING_ALLOCATION --> RUNNING > 2016-02-02 06:21:21,903 [294fb51d-8a4c-c099-dc90-97434056e3d7:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: State to report: RUNNING > 2016-02-02 06:21:48,560 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication. Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087 (user client). Closing connection. > java.io.IOException: syscall:read(...)() failed: Connection reset by peer > 2016-02-02 06:21:48,562 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: State change requested RUNNING --> FAILED > 2016-02-02 06:21:48,562 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-f424-6adc-d668-1659e4353698:0:0: State change requested RUNNING --> FAILED > 2016-02-02 06:21:48,562 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0: State change requested RUNNING --> FAILED > 2016-02-02 06:21:48,563 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51e-5de9-0919-be56-52f75a0532f1:0:0: State change requested RUNNING --> FAILED > 2016-02-02 06:21:48,573 [CONTROL-rpc-event-queue] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-f424-6adc-d668-1659e4353698:0:0: State change requested FAILED --> CANCELLATION_REQUESTED > 2016-02-02 06:21:48,573 [CONTROL-rpc-event-queue] WARN o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-f424-6adc-d668-1659e4353698:0:0: Ignoring unexpected state transition FAILED --> CANCELLATION_REQUESTED > 2016-02-02 06:21:48,580 [CONTROL-rpc-event-queue] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51e-5de9-0919-be56-52f75a0532f1:0:0: State change requested FAILED --> CANCELLATION_REQUESTED > 2016-02-02 06:21:48,580 [CONTROL-rpc-event-queue] WARN o.a.d.e.w.fragment.FragmentExecutor - 294fb51e-5de9-0919-be56-52f75a0532f1:0:0: Ignoring unexpected state transition FAILED --> CANCELLATION_REQUESTED > 2016-02-02 06:21:48,588 [CONTROL-rpc-event-queue] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0: State change requested FAILED --> CANCELLATION_REQUESTED > 2016-02-02 06:21:48,588 [CONTROL-rpc-event-queue] WARN o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0: Ignoring unexpected state transition FAILED --> CANCELLATION_REQUESTED > 2016-02-02 06:21:48,596 [CONTROL-rpc-event-queue] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: State change requested FAILED --> CANCELLATION_REQUESTED > 2016-02-02 06:21:48,596 [CONTROL-rpc-event-queue] WARN o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: Ignoring unexpected state transition FAILED --> CANCELLATION_REQUESTED > 2016-02-02 06:21:48,597 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-f424-6adc-d668-1659e4353698:0:0: State change requested FAILED --> FAILED > 2016-02-02 06:21:48,599 [UserServer-1] WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed channel. Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087 (user client) > io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message. > at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] > 2016-02-02 06:21:48,600 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51e-5de9-0919-be56-52f75a0532f1:0:0: State change requested FAILED --> FAILED > 2016-02-02 06:21:48,600 [UserServer-1] WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed channel. Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087 (user client) > io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message. > at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] > 2016-02-02 06:21:48,601 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0: State change requested FAILED --> FAILED > 2016-02-02 06:21:48,601 [UserServer-1] WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed channel. Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087 (user client) > io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message. > at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] > 2016-02-02 06:21:48,601 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: State change requested FAILED --> FAILED > 2016-02-02 06:21:48,601 [UserServer-1] WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed channel. Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087 (user client) > io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message. > at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] > 2016-02-02 06:21:48,602 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-f424-6adc-d668-1659e4353698:0:0: State change requested FAILED --> FAILED > 2016-02-02 06:21:48,602 [UserServer-1] WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed channel. Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087 (user client) > io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message. > at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] > 2016-02-02 06:21:48,602 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51e-5de9-0919-be56-52f75a0532f1:0:0: State change requested FAILED --> FAILED > 2016-02-02 06:21:48,603 [UserServer-1] WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed channel. Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087 (user client) > io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message. > at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] > 2016-02-02 06:21:48,603 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-c7f6-2c8f-0689-af9de21a6d20:0:0: State change requested FAILED --> FAILED > 2016-02-02 06:21:48,603 [UserServer-1] WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed channel. Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087 (user client) > io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message. > at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] > 2016-02-02 06:21:48,603 [UserServer-1] INFO o.a.d.e.w.fragment.FragmentExecutor - 294fb51d-8a4c-c099-dc90-97434056e3d7:0:0: State change requested FAILED --> FAILED > 2016-02-02 06:21:48,604 [UserServer-1] WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed channel. Connection: /10.10.100.201:31010 <--> /10.10.100.201:45087 (user client) > io.netty.handler.codec.EncoderException: RpcEncoder must produce at least one message. > at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:98) ~[netty-codec-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)