drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pritesh Maker (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (DRILL-5813) A query that includes sort loses Drill connection. Drill sometimes crashes.
Date Mon, 25 Sep 2017 20:53:00 GMT

     [ https://issues.apache.org/jira/browse/DRILL-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pritesh Maker reassigned DRILL-5813:
------------------------------------

    Assignee: Vlad Rozov  (was: Paul Rogers)

> A query that includes sort loses Drill connection.  Drill sometimes crashes.
> ----------------------------------------------------------------------------
>
>                 Key: DRILL-5813
>                 URL: https://issues.apache.org/jira/browse/DRILL-5813
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Relational Operators
>    Affects Versions: 1.11.0
>            Reporter: Robert Hou
>            Assignee: Vlad Rozov
>             Fix For: 1.12.0
>
>         Attachments: drillbit.log
>
>
> Query is:
> {noformat}
> ALTER SESSION SET `exec.sort.disable_managed` = false;
> alter session set `planner.enable_decimal_data_type` = true;
> select count(*) from (select * from dfs.`/drill/testdata/resource-manager/all_types_large`
order by missing11) d where d.missing3 is false;
> {noformat}
> This query has passed before when the number of threads and amount of memory is restricted.
 With more threads and memory, the query does not complete execution.
> Here is the stack trace:
> {noformat}
> Exception occurred with closed channel.  Connection: /10.10.100.190:59281 <-->
/10.10.100.190:31010 (user client)
> java.io.IOException: Connection reset by peer
> 	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> 	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> 	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> 	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
> 	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384)
> 	at oadd.io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311)
> 	at oadd.io.netty.buffer.WrappedByteBuf.setBytes(WrappedByteBuf.java:407)
> 	at oadd.io.netty.buffer.UnsafeDirectLittleEndian.setBytes(UnsafeDirectLittleEndian.java:32)
> 	at oadd.io.netty.buffer.DrillBuf.setBytes(DrillBuf.java:792)
> 	at oadd.io.netty.buffer.MutableWrappedByteBuf.setBytes(MutableWrappedByteBuf.java:280)
> 	at oadd.io.netty.buffer.ExpandableByteBuf.setBytes(ExpandableByteBuf.java:26)
> 	at oadd.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
> 	at oadd.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
> 	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> 	at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> 	at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> User Error Occurred: Connection /10.10.100.190:59281 <--> /10.10.100.190:31010
(user client) closed unexpectedly. Drillbit down?
> oadd.org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Connection /10.10.100.190:59281
<--> /10.10.100.190:31010 (user client) closed un
> expectedly. Drillbit down?
> [Error Id: b97704a4-b8f0-4cd0-b428-2cf1bcf39a1d ]
> 	at oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> 	at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:373)
> 	at oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> 	at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> 	at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> 	at oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> 	at oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> 	at oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
> 	at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
> 	at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
> 	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
> 	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:89)
> 	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> 	at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> 	at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> [#14] Query failed: 
> oadd.org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Connection /10.10.100.190:59281
<--> /10.10.100.190:31010 (user client) closed un
> expectedly. Drillbit down?
> [Error Id: b97704a4-b8f0-4cd0-b428-2cf1bcf39a1d ]
> 	at oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> 	at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:373)
> 	at oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> 	at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> 	at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> 	at oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> 	at oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> 	at oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
> 	at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
> 	at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
> 	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
> 	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:89)
> 	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> 	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> 	at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> 	at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> 	at java.lang.Thread.run(Thread.java:745)
> SYSTEM ERROR: IllegalArgumentException: Attempted to send a message when connection is
no longer valid.
> Query submission to Drillbit failed.
> [Error Id: 72a44d5e-0e4a-4343-ac78-ff4bb1cb1219 ]
> oadd.org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalArgumentException:
Attempted to send a message when connection is no longer va
> lid.
> Query submission to Drillbit failed.
> [Error Id: 72a44d5e-0e4a-4343-ac78-ff4bb1cb1219 ]
> 	at oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> 	at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$SubmissionListener.failed(QueryResultHandler.java:294)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:124)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:91)
> 	at oadd.org.apache.drill.exec.rpc.BasicClient.send(BasicClient.java:174)
> 	at oadd.org.apache.drill.exec.rpc.user.UserClient.submitQuery(UserClient.java:129)
> 	at oadd.org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:794)
> 	at org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:543)
> 	at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1895)
> 	at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
> 	at oadd.org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473)
> 	at org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1100)
> 	at oadd.org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477)
> 	at org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:181)
> 	at oadd.org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109)
> 	at oadd.org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
> 	at org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:112)
> 	at org.apache.drill.test.framework.DrillTestJdbc.executeSetupQuery(DrillTestJdbc.java:181)
> 	at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:139)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: oadd.org.apache.drill.exec.rpc.RpcException: Failure sending message.
> 	... 22 more
> Caused by: java.lang.IllegalArgumentException: Attempted to send a message when connection
is no longer valid.
> 	at oadd.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
> 	at oadd.org.apache.drill.exec.rpc.RequestIdMap.createNewRpcListener(RequestIdMap.java:88)
> 	at oadd.org.apache.drill.exec.rpc.AbstractRemoteConnection.createNewRpcListener(AbstractRemoteConnection.java:162)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:117)
> 	... 21 more
> [#15] Query failed: 
> oadd.org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: IllegalArgumentException:
Attempted to send a message when connection is no longer va
> lid.
> Query submission to Drillbit failed.
> [Error Id: 72a44d5e-0e4a-4343-ac78-ff4bb1cb1219 ]
> 	at oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> 	at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$SubmissionListener.failed(QueryResultHandler.java:294)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:124)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:91)
> 	at oadd.org.apache.drill.exec.rpc.BasicClient.send(BasicClient.java:174)
> 	at oadd.org.apache.drill.exec.rpc.user.UserClient.submitQuery(UserClient.java:129)
> 	at oadd.org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:794)
> 	at org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:543)
> 	at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1895)
> 	at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
> 	at oadd.org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473)
> 	at org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1100)
> 	at oadd.org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477)
> 	at org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:181)
> 	at oadd.org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109)
> 	at oadd.org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:130)
> 	at org.apache.drill.jdbc.impl.DrillStatementImpl.executeQuery(DrillStatementImpl.java:112)
> 	at org.apache.drill.test.framework.DrillTestJdbc.executeSetupQuery(DrillTestJdbc.java:181)
> 	at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:139)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: oadd.org.apache.drill.exec.rpc.RpcException: Failure sending message.
> 	... 22 more
> Caused by: java.lang.IllegalArgumentException: Attempted to send a message when connection
is no longer valid.
> 	at oadd.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
> 	at oadd.org.apache.drill.exec.rpc.RequestIdMap.createNewRpcListener(RequestIdMap.java:88)
> 	at oadd.org.apache.drill.exec.rpc.AbstractRemoteConnection.createNewRpcListener(AbstractRemoteConnection.java:162)
> 	at oadd.org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:117)
> 	... 21 more
> Failed while running cleanup query. Not returning connection to pool.
> org.apache.drill.jdbc.AlreadyClosedSqlException: ResultSet is already closed.
> 	at org.apache.drill.jdbc.impl.DrillResultSetImpl.throwIfClosed(DrillResultSetImpl.java:96)
> 	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:128)
> 	at org.apache.drill.test.framework.DrillTestJdbc.executeSetupQuery(DrillTestJdbc.java:185)
> 	at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:139)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The first error in drillbit.log:
> {noformat}
> 2017-09-22 13:18:51,720 [BitServer-3] WARN  o.a.d.exec.rpc.control.ControlClient - Message
of mode RESPONSE of rpc type 1 took longer than 500ms.  Actual duration was 197783ms.
> 2017-09-22 13:25:51,944 [BitServer-5] WARN  o.a.drill.exec.rpc.data.DataServer - Message
of mode PING of rpc type 0 took longer than 500ms.  Actual duration was 394057ms.
> 2017-09-22 13:38:40,957 [BitServer-5] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception
in RPC communication.  Connection: /10.10.100.190:31012 <--> /10.10.100.190:43721 (data
server).  Closing connection.
> java.lang.OutOfMemoryError: Java heap space
> 2017-09-22 13:38:53,698 [UserServer-1] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception
in RPC communication.  Connection: /10.10.100.190:31010 <--> /10.10.100.190:59281 (user
server).  Closing connection.
> java.lang.OutOfMemoryError: Java heap space
> 2017-09-22 13:42:14,255 [UserServer-1] DEBUG o.a.drill.exec.work.foreman.Foreman - 263aa12c-a069-ef76-02ff-6314ff72dca7:
State change requested RUNNING --> CANCELLATION_REQUESTED
> 2017-09-22 13:46:43,615 [263aa12c-a069-ef76-02ff-6314ff72dca7:frag:2:1] ERROR o.a.drill.common.CatastrophicFailure
- Catastrophic Failure Occurred, exiting. Information message: Unable to handle out of memory
condition in FragmentExecutor.
> java.lang.OutOfMemoryError: Java heap space
> 2017-09-22 13:52:25,011 [Drillbit-ShutdownHook#0] INFO  o.apache.drill.exec.server.Drillbit
- Received shutdown request.
> 2017-09-22 13:52:43,818 [Drillbit-ShutdownHook#0] DEBUG o.apache.drill.exec.server.Drillbit
- Shutdown begun.
> 2017-09-22 13:53:00,684 [BitClient-1] ERROR o.apache.drill.exec.rpc.BasicClient - Unable
to maintain connection /10.10.100.190:43721 <--> qa-node190.qa.lab/10.10.100.190:31012
(data client).  Closing connection.
> {noformat}
> sing cln q47



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message