hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daryn Sharp <da...@oath.com.INVALID>
Subject Re: [DISCUSS] Hadoop RPC encryption performance improvements
Date Wed, 31 Oct 2018 14:39:37 GMT
Various KMS tasks have been delaying my RPC encryption work – which is 2nd
on TODO list.  It's becoming a top priority for us so I'll try my best to
get a preliminary netty server patch (sans TLS) up this week if that helps.

The two cited jiras had some critical flaws.  Skimming my comments, both
use blocking IO (obvious nonstarter).  HADOOP-10768 is a hand rolled
TLS-like encryption which I don't feel is something the community can or
should maintain from a security standpoint.

Daryn

On Wed, Oct 31, 2018 at 8:43 AM Wei-Chiu Chuang <weichiu@apache.org> wrote:

> Ping. Any one? Cloudera is interested in moving forward with the RPC
> encryption improvements, but I just like to get a consensus which approach
> to go with.
>
> Otherwise I'll pick HADOOP-10768 since it's ready for commit, and I've
> spent time on testing it.
>
> On Thu, Oct 25, 2018 at 11:04 AM Wei-Chiu Chuang <weichiu@apache.org>
> wrote:
>
> > Folks,
> >
> > I would like to invite all to discuss the various Hadoop RPC encryption
> > performance improvements. As you probably know, Hadoop RPC encryption
> > currently relies on Java SASL, and have _really_ bad performance (in
> terms
> > of number of RPCs per second, around 15~20% of the one without SASL)
> >
> > There have been some attempts to address this, most notably, HADOOP-10768
> > <https://issues.apache.org/jira/browse/HADOOP-10768> (Optimize Hadoop
> RPC
> > encryption performance) and HADOOP-13836
> > <https://issues.apache.org/jira/browse/HADOOP-13836> (Securing Hadoop
> RPC
> > using SSL). But it looks like both attempts have not been progressing.
> >
> > During the recent Hadoop contributor meetup, Daryn Sharp mentioned he's
> > working on another approach that leverages Netty for its SSL encryption,
> > and then integrate Netty with Hadoop RPC so that Hadoop RPC automatically
> > benefits from netty's SSL encryption performance.
> >
> > So there are at least 3 attempts to address this issue as I see it. Do we
> > have a consensus that:
> > 1. this is an important problem
> > 2. which approach we want to move forward with
> >
> > --
> > A very happy Hadoop contributor
> >
>
>
> --
> A very happy Hadoop contributor
>


-- 

Daryn

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message