Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 0EEC02009EE for ; Wed, 18 May 2016 17:56:15 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 0D9A6160A00; Wed, 18 May 2016 15:56:15 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 325681609B0 for ; Wed, 18 May 2016 17:56:14 +0200 (CEST) Received: (qmail 1314 invoked by uid 500); 18 May 2016 15:56:13 -0000 Mailing-List: contact issues-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list issues@drill.apache.org Received: (qmail 1178 invoked by uid 99); 18 May 2016 15:56:13 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2016 15:56:13 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id D43072C1F5C for ; Wed, 18 May 2016 15:56:12 +0000 (UTC) Date: Wed, 18 May 2016 15:56:12 +0000 (UTC) From: "Khurram Faraaz (JIRA)" To: issues@drill.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (DRILL-3743) query hangs on sqlline once Drillbit on foreman node is killed MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Wed, 18 May 2016 15:56:15 -0000 [ https://issues.apache.org/jira/browse/DRILL-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289192#comment-15289192 ] Khurram Faraaz commented on DRILL-3743: --------------------------------------- A query could have either FAILED, or not completed, or it could be hung or it could be in running state or starting state. I think irrespective of what state a query was in, we should write the query that had started its execution to the profile and the drillbit.log. That way users will get to know if a query was failed/not completed, by seeing the drillbit.log or the query Profiles. I believe we should write the query to the profiles, it is failed we can mark that it has failed or did not complete. > query hangs on sqlline once Drillbit on foreman node is killed > -------------------------------------------------------------- > > Key: DRILL-3743 > URL: https://issues.apache.org/jira/browse/DRILL-3743 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow > Affects Versions: 1.2.0 > Environment: 4 node cluster CentOS > Reporter: Khurram Faraaz > Assignee: Sudheesh Katkam > Priority: Critical > Fix For: 1.7.0 > > > sqlline/query hangs once Drillbit (on Foreman node) is killed. (kill -9 ) > query was issued from the Foreman node. The query returns many records, and it is a long running query. > Steps to reproduce the problem. > set planner.slice_target=1 > 1. clush -g khurram service mapr-warden stop > 2. clush -g khurram service mapr-warden start > 3. ./sqlline -u "jdbc:drill:schema=dfs.tmp" > 0: jdbc:drill:schema=dfs.tmp> select * from `twoKeyJsn.json` limit 2000000; > 4. Immediately from another console do a jps and kill the Drillbit process (in this case foreman) while the query is being run on sqlline. You will notice that sqlline just hangs, we do not see any exceptions or errors being reported on sqlline prompt or in drillbit.log or drillbit.out > I do see this Exception in sqlline.log on the node from where sqlline was started > {code} > 2015-09-04 18:45:12,069 [Client-1] INFO o.a.d.e.rpc.user.QueryResultHandler - User Error Occurred > org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Connection /10.10.100.201:53425 <--> /10.10.100.201:31010 (user client) closed unexpectedly. > [Error Id: ec316cfd-c9a5-4905-98e3-da20cb799ba5 ] > at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:524) ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.rpc.user.QueryResultHandler$SubmissionListener$ChannelClosedListener.operationComplete(QueryResultHandler.java:298) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.DefaultPromise$LateListeners.run(DefaultPromise.java:845) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.DefaultPromise$LateListenerNotifier.run(DefaultPromise.java:873) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > 2015-09-04 18:45:12,069 [Client-1] INFO o.a.d.j.i.DrillResultSetImpl$ResultsListener - [#7] Query failed: > org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Connection /10.10.100.201:53425 <--> /10.10.100.201:31010 (user client) closed unexpectedly. > [Error Id: ec316cfd-c9a5-4905-98e3-da20cb799ba5 ] > at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:524) ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.rpc.user.QueryResultHandler$SubmissionListener$ChannelClosedListener.operationComplete(QueryResultHandler.java:298) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.DefaultPromise$LateListeners.run(DefaultPromise.java:845) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.DefaultPromise$LateListenerNotifier.run(DefaultPromise.java:873) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > 2015-09-04 18:45:12,071 [Client-1] ERROR o.a.d.e.rpc.user.QueryResultHandler - SYSTEM ERROR: ChannelClosedException > [Error Id: c53c477f-f1cf-4458-8620-b1e11ba31701 ] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: ChannelClosedException > [Error Id: c53c477f-f1cf-4458-8620-b1e11ba31701 ] > at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:524) ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.rpc.user.QueryResultHandler$SubmissionListener.failed(QueryResultHandler.java:312) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.rpc.CoordinationQueue$RpcListener.setException(CoordinationQueue.java:103) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.rpc.CoordinationQueue$RpcListener.operationComplete(CoordinationQueue.java:89) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.drill.exec.rpc.CoordinationQueue$RpcListener.operationComplete(CoordinationQueue.java:67) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:788) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:689) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1114) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:705) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:980) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1032) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:965) [netty-transport-4.0.27.Final.jar:4.0.27.Final] > at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) [netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na] > at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) [netty-common-4.0.27.Final.jar:4.0.27.Final] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > Caused by: org.apache.drill.exec.rpc.ChannelClosedException: null > ... 18 common frames omitted > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)