From common-dev-return-102255-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Mon Jun 3 19:56:38 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id CE9DB18062F for ; Mon, 3 Jun 2019 21:56:37 +0200 (CEST) Received: (qmail 85775 invoked by uid 500); 3 Jun 2019 19:56:36 -0000 Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-dev@hadoop.apache.org Received: (qmail 85753 invoked by uid 99); 3 Jun 2019 19:56:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jun 2019 19:56:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 3EC5A181255; Mon, 3 Jun 2019 19:56:35 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.495 X-Spam-Level: * X-Spam-Status: No, score=1.495 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.305, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id roN0koKbsix3; Mon, 3 Jun 2019 19:56:33 +0000 (UTC) Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 878A45F169; Mon, 3 Jun 2019 19:56:32 +0000 (UTC) Received: by mail-oi1-f178.google.com with SMTP id b20so10223252oie.12; Mon, 03 Jun 2019 12:56:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=pOmxzQlwuBsQmsFWuTgwY2LscBik2n4Wgwo8A5dCswk=; b=XjA54wnZWnaZQWjKo36WwyMchCQDl2Euep1shlvr8y2wSq4pbsXklLgEex4DNUxCfB 46jy5jty8G+x9I4kuvJmA1RDEPWmjHp0B0CAE+7tYS0Bvjd4/XGqyecMXipHdLpF+wmF npZ705jx+yXZDK+QnlpPhANB4IX8Ao6fjiFBaS2kly48r6X6Jbo78zj7DOG5myI66I6O Zg+CahXPIXhHnOgw3B9seg3vg1Xg18WbgmDs5lSOTgxLN17wy88ljieccalL8VE8+D9T 8lhrmhBZDueg7U75eEKI9QAFmSG4G4yhDqdFy2IxdsThjwwbpGWPv5Rx6rZhkNWAXfw8 tp3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=pOmxzQlwuBsQmsFWuTgwY2LscBik2n4Wgwo8A5dCswk=; b=sxVTiQhDKL+bApUaUCAqM09tHyDa9lUfnpwzn+8JbXoWG5b9CUNbTMq3HQQiaQMZr6 UrN3L+NzeBs4UhxlmWEc2F/raIbD5ZASLeCO//sy9ePaZNuf/GRzSnndadVhMT/+qHs1 KUTrF0Wsjf5GqyvOj8+YI8eX/eEEMvR0pgY9qvGl+RloQN9eP0rGuEaBKU0xn3SdQhDJ 3epzzBJuT/7p7S54qZZBAqF5Ndas6UNTRj0/CpDyYPvtQkFWxmvdQ0VnC2prcE+2viGY LhAGu42giHBuVafwfRKfnRhpkgm+g+uzoy2pHbs2xEO3v+gLXtJyk1A5VdbY9QNtKVSr DEHA== X-Gm-Message-State: APjAAAX/9mUmr/m5MjgcATgGI/xHbcaIfSrlY1zHokWJgWtUgxsPXY/w 6zVKfNNQyPbVJYW1Najlj5eesHZuBKfwtlV6WXk= X-Google-Smtp-Source: APXvYqxS+H8kSKVnA02L/+DfVP4ObvhKcZ7WTV+osvcNKfJWCa0rWR2nl9DaNkM89EZZ7GkeXqzOL3foB8GUo0XD/7Q= X-Received: by 2002:aca:b344:: with SMTP id c65mr2161396oif.46.1559591791186; Mon, 03 Jun 2019 12:56:31 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Giovanni Matteo Fumarola Date: Mon, 3 Jun 2019 12:56:20 -0700 Message-ID: Subject: Re: [DISCUSS] Merge HDFS-13891(RBF) to trunk To: =?UTF-8?B?ScOxaWdvIEdvaXJp?= Cc: Akira Ajisaka , Brahma Reddy Battula , Hdfs-dev , Hadoop Common Content-Type: multipart/alternative; boundary="000000000000b5588e058a70c2d4" --000000000000b5588e058a70c2d4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable +1 on merging. Thanks Brahma for starting the thread. On Mon, Jun 3, 2019 at 10:00 AM I=C3=B1igo Goiri wrote: > Thank you Brahma for pushing this. > > As you mentioned, we have already taken most of the changes into > production. > I want to highlight that the main contribution is the addition of securit= y. > We have been able to test this at a smaller scale (~500 servers and 4 > subclusters) and the performance is great with our current ZooKeeper > deployment. > I would also like to highlight that all the changes are constrained to > hadoop-hdfs-rbf and there is no differences in commons or HDFS. > > +1 on merging > > Inigo > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka wrote: > > > Thanks Brahma for starting the discussion. > > I'm +1 for merging this. > > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster > > with 2 routers (not in production) and running several tests. > > > > Regards, > > Akira > > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula > > wrote: > > > > > > Dear Hadoop Developers > > > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk. We > have > > > been working on this feature from last several months. > > > This feature work received the contributions from different companies= . > > All > > > of the feature development happened smoothly and collaboratively in > > JIRAs. > > > > > > Kindly do take a look at the branch and raise issues/concerns that ne= ed > > to > > > be addressed before the merge. > > > > > > *Highlights of HDFS-13891 Branch:* > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D > > > > > > Adding Security to RBF(1) > > > Adding Missing Client API's(2) > > > Improvements/Bug Fixing > > > Critical - HDFS-13637, HDFS-13834 > > > > > > *Commits:* > > > =3D=3D=3D=3D=3D=3D=3D=3D > > > > > > No of JIRAs Resolved: 72 > > > > > > All this commits are in RBF Module. No changes in hdfs/common. > > > > > > *Tested Cluster:* > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > Most of these changes verified at Uber,Microsoft,Huawei and some othe= r > > > compaines. > > > > > > *Uber*: Most changes are running in production @Uber including the > > critical > > > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS Routers. > > > Zookeeper as a state store to hold delegation tokens were also stress > > > tested to hold more than 2 Million tokens. --CR Hota > > > > > > *MicroSoft*: Most of these changes are currently running in productio= n > at > > > Microsoft.The security has also been tested in a 500 server cluster > with > > 4 > > > subclsuters. --Inigo Goiri > > > > > > *Huawei* : Deployed all this changes in 20 node cluster with 3 > > > routers.Planning deploy 10K production cluster. > > > > > > *Contributors:* > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu Asanuma,Shubham > > > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang > > > Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He > Xiaoqiao,Sherwood > > > Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving discussion= s > > and > > > contributing to this. > > > > > > *Future Tasks:* > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > will cleanup the jira's under this umbrella and contiue to work. > > > > > > Reference: > > > 1) https://issues.apache.org/jira/browse/HDFS-13532 > > > 2) https://issues.apache.org/jira/browse/HDFS-13655 > > > > > > > > > > > > > > > --Brahma Reddy Battula > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org > > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org > > > > > --000000000000b5588e058a70c2d4--