drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Parth Chandra <par...@apache.org>
Subject Re: Upgrading Netty
Date Fri, 19 May 2017 21:29:50 GMT
Looks like the specific issue I was referring to was addressed in Netty
4.0.29 [1]. The comment for the commit is a little concerning:

<quote>

Result:

ThreadPoolCache is now also usable and so gives performance
improvements when allocation and deallocation thread are different.

Performance when using same thread for allocation and deallocation is
noticable worse then before.


</quote>


We might want to do a performance run to make sure things are no worse than
before.


[1]
https://github.com/netty/netty/commit/f765053ae740e300a6b696840d7dfe5de32afeb3


On Mon, May 15, 2017 at 5:46 PM, Parth Chandra <parthc@apache.org> wrote:

> The per thread allocation cache in Netty causes unbounded memory growth in
> Drill because we pass the ownership of a buffer from one thread to another.
> The version we use has a fix for the Drill use case where Netty will no
> longer add a buffer to its per thread cache if the buffer was allocated by
> a thread which is different from the thread freeing the buffer.
> This fix was reversed in a subsequent release and the latest version has
> the same issue.
> There might have been a fix in Netty for this in some other place which I
> am not aware of (perhaps they removed it altogether as Paul seems to have
> seen).
> AFAIK, we do not have a direct reference to that code in Drill's
> allocator. If you try to upgrade and hit an issue, post it here.
> If you are able to upgrade the Netty version, then run a longevity test to
> make sure there is no 'leaking' of memory from one thread to another.
>
>
>
> On Mon, May 15, 2017 at 4:08 PM, Paul Rogers <progers@mapr.com> wrote:
>
>> As it turns out, Drill makes clever use of the internal details of the
>> Netty memory allocator. But, that code changed significantly in the last
>> couple of years. When I attempted to upgrade, I found that the private
>> features of Netty that the Drill allocator uses no longer exist in the
>> latest Netty.
>>
>> So, someone will need to understand what that part of the Drill allocator
>> does and design an alternative integration.
>>
>> The particular issue seems to be that Netty had a per-thread allocation
>> cache which seems to not exist in the latest version.
>>
>> - Paul
>>
>> > On May 15, 2017, at 3:58 PM, Sudheesh Katkam <skatkam@mapr.com> wrote:
>> >
>> > Hi all,
>> >
>> > As part of working on DRILL-5431 [1], I found a bug in Netty [2], which
>> is due to be fixed in 4.0.48 [3]. Drill is currently using 4.0.27 [4]. Does
>> anyone foresee issues with upgrading to the latest version of Netty? I
>> noticed Apache Arrow upgraded to 4.0.41 [5].
>> >
>> > Thank you,
>> > Sudheesh
>> >
>> > [1] https://issues.apache.org/jira/browse/DRILL-5431
>> > [2] https://github.com/netty/netty/issues/6709
>> > [3] https://github.com/netty/netty/pull/6713
>> > [4] https://github.com/apache/drill/blob/master/pom.xml#L550
>> > [5] https://github.com/apache/arrow/commit/3487c2f0cdc2297a80ba3
>> 525c192745313b3da48
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message