drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@apache.org>
Subject Re: connection allocator in rpc layer is using too much memory
Date Tue, 07 Jul 2015 17:02:28 GMT
There is a time where data is read off the socket before we know what type
of message it is.  This socket read buffer is outside the normal flow and
could grow (although it shouldn't get this big).  However, the memory
you're talking about here is memory allocated due to the size of the
incoming message.  My guess would be either you have unusually large
records or the length of the message being sent was corrupted.  (Assuming
you are talking about the allocation at [1]).

I would start logging unusually large record batches and see if something
weird is going on.  A record batch shouldn't be larger than 65k records so
for the batch to be 1gb in size would require each record to be 16k in size
and for the batch to be the maximum number of records.  More realistically,
we generally target 4k records in a batch which would suggest records that
are 256k.

[1]
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/ProtobufLengthDecoder.java#L87

On Tue, Jul 7, 2015 at 9:13 AM, Abdel Hakim Deneche <adeneche@maprtech.com>
wrote:

> Trying to investigate DRILL-3241
> <https://issues.apache.org/jira/browse/DRILL-3241> (query hangs if out of
> memory in RPC layer), I see the following warning in the logs:
>
> WARN: Failure allocating buffer on incoming stream due to
>
>  memory limits.  Current Allocation: 1372678764.
>
>
> This happening in ProtobufLengthDecoder.decode() on the receiver side (data
> server).
>
> Is it expected for the connection allocation to allocation > 1GB of memory
> ? shouldn't the allocated batches be transferred to the receiving
> fragment's allocator ?
>
> Thanks!
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message