Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 3 Feb 2017 18:30:52 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13025717.1480967499000.28266.1486146652012@Atlassian.JIRA>
In-Reply-To: <JIRA.13025717.1480967499000@Atlassian.JIRA>
References: <JIRA.13025717.1480967499000@Atlassian.JIRA> <JIRA.13025717.1480967499170@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 03 Feb 2017 18:30:59 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851878#comment-15851878 ] 

stack commented on HADOOP-13866:
--------------------------------

I tried the patch here against the hadoop 2.7 branch and it fixed my netty issue running hbase master.

+1 on commit to trunk and hadoop3 branch.

Can we get it in 2.8 branch too? How can I help?

It would be obnoxious asking for it to be committed to 2.7. As is hbase-2 won't work with anything without this patch (unless hbase shades its netty). I'll ask anyways.

Here is some background on how the incompatibillity came in, in case it helps:

The clashing io.netty 4 (4.0.23.Final) gets added to hadoop with below commit that lands before the release of hadoop-2.7.0.
commit bbdd990864d677e99b8fc73bdf720d66e2187d2c
Author: Haohui Mai <wheat9@apache.org>
Date: Tue Oct 28 16:53:53 2014 -0700
HDFS-7280. Use netty 4 in WebImageViewer. Contributed by Haohui Mai.

The hbase commit that moves us from a compatible 4.0.30 to an incompatible 4.1.1 is:
commit bd45cf34762332a3a51f605798a3e050e7a1e62e
Author: Jurriaan Mous <jurmous@jurmo.us>
Date: Fri Jun 10 17:57:42 2016 +0200
HBASE-16004 Update to Netty 4.1.1
Signed-off-by: stack <stack@apache.org>

Maybe we could go back but we will be leaning heavily on netty by the time 2.0 ships (It is basis of new async client, it is used in our async dfsclient implementation -- hopefully moved back up to hdfs at some point but for now living in hbase -- and it is looking like netty will be our new rpc chassis too by the time 2.0 ships).

hadoop-2.6.x doesn't have HDFS-7280 commit so hbase-2.0.0 should work with it, but not later versions of hadoop.

I can test other combos if that will help get stuff committed. Just say. Thanks.

> Upgrade netty-all to 4.1.1.Final
> --------------------------------
>
>                 Key: HADOOP-13866
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13866
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>            Priority: Blocker
>         Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch, HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch, HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/  usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_000005/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1] io.netty.util.concurrent.DefaultPromise: An exception was thrown by org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError: io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
>         at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
>         at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
>         at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
>         at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
>         at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
>         at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org