drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3317) when ProtobufLengthDecoder couldn't allocate a new DrillBuf, this error is just logged and nothing else is done
Date Wed, 20 Apr 2016 15:56:25 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15250135#comment-15250135
] 

ASF GitHub Bot commented on DRILL-3317:
---------------------------------------

Github user adeneche commented on the pull request:

    https://github.com/apache/drill/pull/446#issuecomment-212488188
  
    So I spent more time looking at ways to create a unit test for this, and I don't think
it's possible:
    First, current implementation of ProtobufLengthDecoder for the data tunnel uses the root
allocator, which is weird and changing the rpc max allocation won't help in this case.
    Second, even after I changed the implementation and made sure ProtobufLengthDecoder uses
the rpc allocator, this same allocator is used by netty too and you can't control where it
will run out of memory, in most of my experiments it run out of memory in ByteToMessageDecoder,
so writing a unit test using this won't be reliable as it won't consistently reproduce the
issue.



> when ProtobufLengthDecoder couldn't allocate a new DrillBuf, this error is just logged
and nothing else is done
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: DRILL-3317
>                 URL: https://issues.apache.org/jira/browse/DRILL-3317
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - RPC
>            Reporter: Deneche A. Hakim
>            Assignee: Jacques Nadeau
>             Fix For: 1.7.0
>
>
> Trying to reproduce DRILL-3241 I sometimes get the following error in the logs:
> {noformat}
> ERROR: Out of memory outside any particular fragment.
> 	at org.apache.drill.exec.rpc.data.DataResponseHandlerImpl.informOutOfMemory(DataResponseHandlerImpl.java:40)
> 	at org.apache.drill.exec.rpc.data.DataServer$2.handle(DataServer.java:227)
> 	at org.apache.drill.exec.rpc.ProtobufLengthDecoder.decode(ProtobufLengthDecoder.java:87)
> 	at org.apache.drill.exec.rpc.data.DataProtobufLengthDecoder$Server.decode(DataProtobufLengthDecoder.java:52)
> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
> 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> WARN: Failure allocating buffer on incoming stream due to memory limits.  Current Allocation:
1372678764.
> 	at org.apache.drill.exec.rpc.ProtobufLengthDecoder.decode(ProtobufLengthDecoder.java:85)
> 	at org.apache.drill.exec.rpc.data.DataProtobufLengthDecoder$Server.decode(DataProtobufLengthDecoder.java:52)
> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
> 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> 	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> {noformat}
> ProtobufLengthDecoder.decode() does call OutOfMemoryHandler.handle() which calls DataResponseHandlerImpl.informOutOfMemory()
which just logs the error in the logs.
> If we have fragments waiting for data they will be stuck waiting forever, and the query
will hang (behavior observed in DRILL-3241



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message