drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacques Nadeau (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (DRILL-2750) Running 1 or more queries against Drillbits having insufficient DirectMem renders the Drillbits in an unusable state
Date Fri, 15 May 2015 07:08:01 GMT

     [ https://issues.apache.org/jira/browse/DRILL-2750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jacques Nadeau resolved DRILL-2750.
-----------------------------------
    Resolution: Fixed

Resolved by stability patches over last few days.

> Running 1 or more queries against Drillbits having insufficient DirectMem renders the
Drillbits in an unusable state
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: DRILL-2750
>                 URL: https://issues.apache.org/jira/browse/DRILL-2750
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Flow
>    Affects Versions: 0.9.0
>         Environment: RHEL 6.4
>            Reporter: Kunal Khatua
>            Assignee: Chris Westin
>            Priority: Critical
>             Fix For: 1.0.0
>
>         Attachments: DRILL-2750.1.patch.txt
>
>
> When running queries against a Drill cluster with limited DirectMem; if one or more queries
fail due to insufficient memory, then even queries that should easily run within the allocated
memory fail.
> The initial failure when queries with large memory requirements fail: 
> 2015-04-10 09:57:55 [pip0] ERROR PipSQuawkling fetchRows - [ 1 / 16_par1000 ] Failure
while executing query.
> java.sql.SQLException: Failure while executing query.
>         at org.apache.drill.jdbc.DrillCursor.next(DrillCursor.java:144)
>         at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
>         at org.apache.drill.jdbc.DrillResultSet.next(DrillResultSet.java:85)
>         at PipSQuawkling.fetchRows(PipSQuawkling.java:319)
>         at PipSQuawkling.executeTest(PipSQuawkling.java:154)
>         at PipSQuawkling.run(PipSQuawkling.java:76)
> Caused by: org.apache.drill.exec.rpc.RpcException: RemoteRpcException: Failure while
running fragment.[ e8c657a7-93a9-415a-8641-a4fbd4836a65 on ucs-node5.perf.lab:31010 ]
> [ e8c657a7-93a9-415a-8641-a4fbd4836a65 on ucs-node5.perf.lab:31010 ]
>         at org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:111)
>         at org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:100)
>         at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:52)
>         at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:34)
>         at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:57)
>         at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:194)
>         at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:173)
>         at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>         at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161)
>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>         at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
>         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
>         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>         at java.lang.Thread.run(Thread.java:744)
> After that, subsequent queries that should run, fail with the following:
> 2015-04-10 09:59:29 [pip0] ERROR PipSQuawkling executeQuery - [ 2 / rerun_06_par1000
] exception while executing query: Failure while executing query.
> java.sql.SQLException: exception while executing query: Failure while executing query.
>         at net.hydromatic.avatica.Helper.createException(Helper.java:40)
>         at net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:406)
>         at net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351)
>         at net.hydromatic.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:78)
>         at PipSQuawkling.executeQuery(PipSQuawkling.java:284)
>         at PipSQuawkling.executeTest(PipSQuawkling.java:144)
>         at PipSQuawkling.run(PipSQuawkling.java:76)
> Caused by: java.sql.SQLException: Failure while executing query.
>         at org.apache.drill.jdbc.DrillCursor.next(DrillCursor.java:144)
>         at org.apache.drill.jdbc.DrillResultSet.execute(DrillResultSet.java:105)
>         at org.apache.drill.jdbc.DrillResultSet.execute(DrillResultSet.java:44)
>         at net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404)
>         ... 5 more
> Caused by: org.apache.drill.exec.rpc.RpcException: RemoteRpcException: Failure while
trying to start remote fragment, You attempted to create a new child allocator with initial
reservation 6000000 but only 110395 bytes of memory were available. [ 689006cb-d703-42c3-860d
> -bfecc0a66312 on ucs-node10.perf.lab:31010 ]
>         at org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:111)
>         at org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:100)
>         at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:52)
>         at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:34)
>         at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:57)
>         at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:194)
>         at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:173)
>         at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>         at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161)
>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>         at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
>         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
>         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>         at java.lang.Thread.run(Thread.java:744)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message