hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From IƱigo Goiri <elgo...@gmail.com>
Subject Re: [DISCUSS] Merge HDFS-13891(RBF) to trunk
Date Mon, 03 Jun 2019 16:59:55 GMT
Thank you Brahma for pushing this.

As you mentioned, we have already taken most of the changes into production.
I want to highlight that the main contribution is the addition of security.
We have been able to test this at a smaller scale (~500 servers and 4
subclusters) and the performance is great with our current ZooKeeper
deployment.
I would also like to highlight that all the changes are constrained to
hadoop-hdfs-rbf and there is no differences in commons or HDFS.

+1 on merging

Inigo

On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka <aajisaka@gmail.com> wrote:

> Thanks Brahma for starting the discussion.
> I'm +1 for merging this.
>
> FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster
> with 2 routers (not in production) and running several tests.
>
> Regards,
> Akira
>
> On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula <brahma@apache.org>
> wrote:
> >
> > Dear Hadoop Developers
> >
> > I would like to propose RBF Branch (HDFS-13891) merge into trunk. We have
> > been working on this feature from last several months.
> > This feature work received the contributions from different companies.
> All
> > of the feature development happened smoothly and collaboratively in
> JIRAs.
> >
> > Kindly do take a look at the branch and raise issues/concerns that need
> to
> > be addressed before the merge.
> >
> > *Highlights of HDFS-13891 Branch:*
> > =============================
> >
> > Adding Security to RBF(1)
> > Adding Missing Client API's(2)
> > Improvements/Bug Fixing
> >   Critical - HDFS-13637, HDFS-13834
> >
> > *Commits:*
> > ========
> >
> > No of JIRAs Resolved: 72
> >
> > All this commits are in RBF Module. No changes in hdfs/common.
> >
> > *Tested Cluster:*
> > =============
> >
> > Most of these changes verified at Uber,Microsoft,Huawei and some other
> > compaines.
> >
> > *Uber*: Most changes are running in production @Uber including the
> critical
> > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS Routers.
> > Zookeeper as a state store to hold delegation tokens were also stress
> > tested to hold more than 2 Million tokens. --CR Hota
> >
> > *MicroSoft*: Most of these changes are currently running in production at
> > Microsoft.The security has also been tested in a 500 server cluster with
> 4
> > subclsuters. --Inigo Goiri
> >
> > *Huawei* : Deployed all this changes in 20 node cluster with 3
> > routers.Planning deploy 10K production cluster.
> >
> > *Contributors:*
> > ===========
> >
> > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu Asanuma,Shubham
> > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang
> > Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He Xiaoqiao,Sherwood
> > Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving discussions
> and
> > contributing to this.
> >
> > *Future Tasks:*
> > ============
> >
> >  will cleanup the jira's under this umbrella and contiue to work.
> >
> > Reference:
> > 1) https://issues.apache.org/jira/browse/HDFS-13532
> > 2) https://issues.apache.org/jira/browse/HDFS-13655
> >
> >
> >
> >
> > --Brahma Reddy Battula
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message