hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gour Saha <gs...@hortonworks.com>
Subject Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Date Thu, 05 Apr 2018 19:51:21 GMT
Sorry for the really late send.

+1 (non-binding)

-Verified checksums and sha1
-Built from source and ran YARN Service core and api tests successfully
-Installed in a 7-node cluster
-Launched, flexed up/down, stopped, started and destroyed the sample sleeper service
-Launched, flexed, stopped, started Hive/LLAP
-Tested the Service REST APIs and command line

-Gour

On 4/5/18, 10:49 AM, "Wangda Tan" <wheeleast@gmail.com> wrote:

    Thanks everybody for voting!
    The vote passes successfully with 11 binding +1 votes, 8 non-binding +1
    votes and no -1s.
    
    I will work on the staging and releases.
    
    Best,
    Wangda Tan
    
    
    On Thu, Apr 5, 2018 at 10:46 AM, Vinod Kumar Vavilapalli <vinodkv@apache.org
    > wrote:
    
    > That is a great observation. And I missed your previous email about the
    > shaded vs unshaded jars already getting fixed.
    >
    > I guess we are good to go.
    >
    > ------------------------------------------------------------
    > --------------------------------------------------------------
    >
    > Looking at the RC. Went through my usual check-list. Here's my summary.
    >
    > Verification
    > - [Check] Successful recompilation from source tar-ball
    > - [Check] Signature verification
    >     -- Note: The format of the mds files changed a bit - not a biggie.
    >     -- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "
    > *hadoop-3.0.0-src.tar.gz*: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279
    > 7BCED50B B2600318 B7E0BE3A 74DFFF71"
    >     -- But in 3.1.0 RC it is, "
    > */build/source/target/artifacts/hadoop-3.1.0.tar.gz*: SHA256 = 670D2CED
    > 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
    > - [Check] Generating dist tarballs from source tar-ball
    > - [Check] Testing
    >    -- Start NN, DN, RM, NM, JHS, Timeline Service
    >    -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
    > grep, pi
    >    -- Tested CLIs to print nodes, apps etc and also navigated UIs
    >
    > +1 binding.
    >
    > Thanks
    > +Vinod
    >
    > On Apr 3, 2018, at 8:13 PM, Wangda Tan <wheeleast@gmail.com> wrote:
    >
    > Hi Vinod / Arpit,
    >
    > I checked following versions:
    > - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
    >
    > Jars in maven repo [1] are *always* different from jars in the binary
    >
    > tarball [2]: (I only checked hadoop-yarn-api-version.jar)
    >
    > (Following numbers are sizes of the jar)
    > 2.6.5:
    > - Jar in Maven: 1896185
    > - Jar in tarball: 1891485
    >
    > 2.7.5:
    > - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
    > - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
    > (Even size is same, md5 is different)
    >
    > 2.8.3:
    > - Jar in Maven: 2451433
    > - Jar in tarball: 2438975
    >
    > 2.9.0:
    > - Jar in Maven: 2791477
    > - Jar in tarball: 2777789
    >
    > 3.0.1:
    > - Jar in Maven: 2852604
    > - Jar in tarball: 2851373
    >
    > I guess the differences come from our release process.
    >
    > Thanks,
    > Wangda
    >
    > [1] Maven jars are downloaded from
    > https://repository.apache.org/service/local/repositories/
    > releases/content/org/apache/hadoop/hadoop-yarn-api/
    > <version>/hadoop-yarn-api-<version>.jar
    > [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
    >
    >
    > On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
    > vinodkv@apache.org>
    > wrote:
    >
    > We vote on the source code. The binaries are convenience artifacts.
    >
    > This is what I would do - (a) Just replace both the maven jars as well as
    > the binaries to be consistent and correct. And then (b) Give a couple more
    > days for folks who tested on the binaries to reverify - I count one such
    > clear vote as of now.
    >
    > Thanks
    > +Vinod
    >
    >
    > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com> wrote:
    >
    > HI Arpit,
    >
    > I think it won't match if we do rebuild. It should be fine as far as
    > they're signed, correct? I don't see any policy doesn't allow this.
    >
    > Thanks,
    > Wangda
    >
    >
    > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagarwal@hortonworks.com>
    > wrote:
    >
    > Thanks Wangda, I see the shaded jars now.
    >
    > Are the repo jars required to be the same as the binary release? They
    > don’t match right now, probably they got rebuilt.
    >
    > +1 (binding), modulo that remaining question.
    >
    > * Verified signatures
    > * Verified checksums for source and binary artefacts
    > * Sanity checked jars on r.a.o.
    > * Built from source
    > * Deployed to 3 node secure cluster with NameNode HA
    > * Verified HDFS web UIs
    > * Tried out HDFS shell commands
    > * Ran sample MapReduce jobs
    >
    > Thanks!
    >
    >
    > ----------------------------------------------------------------------
    > From: Wangda Tan <wheeleast@gmail.com>
    > Date: Monday, April 2, 2018 at 9:25 PM
    > To: Arpit Agarwal <aagarwal@hortonworks.com>
    > Cc: Gera Shegalov <gera05@gmail.com>, Sunil G <sunilg@apache.org>, "
    > yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>, Hdfs-dev <
    > hdfs-dev@hadoop.apache.org>, Hadoop Common <common-dev@hadoop.apache.org>,
    > "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
    > Vinod Kumar Vavilapalli <vinodkv@apache.org>
    > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
    >
    > As pointed by Arpit, the previously deployed shared jars are incorrect.
    > Just redeployed jars and staged. @Arpit, could you please check the updated
    > Maven repo? https://repository.apache.org/content/repositories/
    > orgapachehadoop-1092
    >
    > Since the jars inside binary tarballs are correct (
    > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
    > need roll another RC, just update Maven repo should be sufficient.
    >
    > Best,
    > Wangda
    >
    >
    > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com>
    > wrote:
    > Hi Arpit,
    >
    > Thanks for pointing out this.
    >
    > I just removed all .md5 files from artifacts. I found md5 checksums still
    > exist in .mds files and I didn't remove them from .mds file because it is
    > generated by create-release script and Apache guidance is "should not"
    > instead of "must not". Please let me know if you think they need to be
    > removed as well.
    >
    > - Wangda
    >
    >
    >
    > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
    > aagarwal@hortonworks.com> wrote:
    > Thanks for putting together this RC, Wangda.
    >
    > The guidance from Apache is to omit MD5s, specifically:
    >
    > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
    >
    >
    > https://www.apache.org/dev/release-distribution#sigs-and-sums
    >
    >
    >
    >
    > On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheeleast@gmail.com>
    > wrote:
    >
    > Hi Gera,
    >
    > It's my bad, I thought only src/bin tarball is enough.
    >
    > I just uploaded all other things under artifact/ to
    > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
    >
    > Please let me know if you have any other comments.
    >
    > Thanks,
    > Wangda
    >
    >
    > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:gera05@gmail.com>
    > wrote:
    >
    >
    > Thanks, Wangda!
    >
    > There are many more artifacts in previous votes, e.g., see
    > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
    > site tarball is missing.
    >
    > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sunilg@apache.org> wrote:
    >
    >
    > Thanks Wangda for initiating the release.
    >
    > I tested this RC built from source file.
    >
    >
    >  - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
    > UI.
    >  - Below feature sanity is done
    >     - Application priority
    >     - Application timeout
    >     - Intra Queue preemption with priority based
    >     - DS based affinity tests to verify placement constraints.
    >  - Tested basic NodeLabel scenarios.
    >     - Added couple of labels to few of nodes and behavior is coming
    >     correct.
    >     - Verified old UI  and new YARN UI for labels.
    >     - Submitted apps to labelled cluster and it works fine.
    >     - Also performed few cli commands related to nodelabel.
    >  - Test basic HA cases and seems correct.
    >  - Tested new YARN UI . All pages are getting loaded correctly.
    >
    >
    > - Sunil
    >
    > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com>
    > wrote:
    >
    >
    > Hi folks,
    >
    > Thanks to the many who helped with this release since Dec 2017 [1].
    > We've
    >
    > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
    >
    > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
    >
    > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
    > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
    >
    > The maven artifacts are available via http://repository.apache.org at
    > https://repository.apache.org/content/repositories/
    > orgapachehadoop-1090/
    >
    > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
    >
    > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
    > include the first class GPU/FPGA support on YARN, Native services,
    > Support
    >
    > rich placement constraints in YARN, S3-related enhancements, allow HDFS
    > block replicas to be provided by an external storage system, etc.
    >
    > For 3.1.0 RC0 vote discussion, please see [3].
    >
    > We’d like to use this as a starting release for 3.1.x [1], depending on
    > how
    >
    > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
    > as
    >
    > the stable release.
    >
    > We have done testing with a pseudo cluster:
    > - Ran distributed job.
    > - GPU scheduling/isolation.
    > - Placement constraints (intra-application anti-affinity) by using
    > distributed shell.
    >
    > My +1 to start.
    >
    > Best,
    > Wangda/Vinod
    >
    > [1]
    >
    > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
    > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org
    > <http://3cyarn-dev.hadoop.apache.org/>%3E
    >
    > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
    > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
    > BY
    >
    > fixVersion ASC
    > [3]
    >
    > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
    > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org
    > <http://3cyarn-dev.hadoop.apache.org/>%3E
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    >
    


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Mime
View raw message