hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shane Kumpf <shane.kumpf.apa...@gmail.com>
Subject Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Date Thu, 05 Apr 2018 14:23:08 GMT
Thanks for putting together the RC, Wangda! Sorry for the late addition.

+1 (non-binding)

- Verified checksums and signatures of the artifacts
- Deployed a pseudo distributed cluster using the binary tgz, source tgz,
and git tag
- Ran basic hdfs commands
- Ran pi, sleep, distributed shell, and a sleeper service using the default
and docker runtimes
- Ran dshell using the default and docker runtimes in a secure cluster
- Verified docker runtime and privileged containers can be disabled
- Verified YARN and HDFS UI's
- Validated the change log against the commits, all discrepancies are
documented or expected
- Basic validation of Registry DNS lookups

On Thu, Apr 5, 2018 at 12:26 AM, Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:

> +1 (binding)
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Built from source
> * Deployed a pseudo Hadoop cluster
> * Verified YARN & HDFS web UIs
> * Ran sample MapReduce jobs
> * Tested NodeLabels
>
> Regards,
> + Naga
>
> On Thu, Apr 5, 2018 at 2:59 AM, Suma Shivaprasad <
> sumasai.shivaprasad@gmail.com> wrote:
>
> > +1 (non binding)
> >
> >
> > *Verified - User Group Queue mapping - Node labels with New UI- Dynamic
> > queuesThanksSuma*
> >
> >
> > On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran <stevel@hortonworks.com>
> > wrote:
> >
> > > that's "dangerously interesting". I think you are right, and I also
> think
> > > it'll just be the version files which get generated
> > >
> > > anyway, +1 binding
> > >
> > > * ran my new Hadoop-3 profile on spark (SPARK-23807), with the
> committer
> > > binding, then my downstream tests. All is well, provided you also have
> a
> > > spark hive JAR patched to accept hadoop 3 as a legitimate hadoop
> version.
> > > That's an ongoing issue in the Spark project. With that JAR on my CP my
> > > downstream tests were all happy (yesterday)
> > >
> > > * today the staging files seem to be missing, at least maven is unable
> to
> > > find them even when I turn the spark snapshots-and-staging profile on.
> > > That'll be the maven dist process at play, nothing else
> > >
> > > On 4 Apr 2018, at 04:13, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > > eeleast@gmail.com>> wrote:
> > >
> > > Hi Vinod / Arpit,
> > >
> > > I checked following versions:
> > > - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
> > >
> > > Jars in maven repo [1] are *always* different from jars in the binary
> > > tarball [2]: (I only checked hadoop-yarn-api-version.jar)
> > >
> > > (Following numbers are sizes of the jar)
> > > 2.6.5:
> > > - Jar in Maven: 1896185
> > > - Jar in tarball: 1891485
> > >
> > > 2.7.5:
> > > - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> > > - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> > > (Even size is same, md5 is different)
> > >
> > > 2.8.3:
> > > - Jar in Maven: 2451433
> > > - Jar in tarball: 2438975
> > >
> > > 2.9.0:
> > > - Jar in Maven: 2791477
> > > - Jar in tarball: 2777789
> > >
> > > 3.0.1:
> > > - Jar in Maven: 2852604
> > > - Jar in tarball: 2851373
> > >
> > > I guess the differences come from our release process.
> > >
> > > Thanks,
> > > Wangda
> > >
> > > [1] Maven jars are downloaded from
> > > https://repository.apache.org/service/local/repositories/
> > > releases/content/org/apache/hadoop/hadoop-yarn-api/
> > > <version>/hadoop-yarn-api-<version>.jar
> > > [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/
> > common/
> > >
> > >
> > > On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> > > vinodkv@apache.org<mailto:vinodkv@apache.org>>
> > > wrote:
> > >
> > > We vote on the source code. The binaries are convenience artifacts.
> > >
> > > This is what I would do - (a) Just replace both the maven jars as well
> as
> > > the binaries to be consistent and correct. And then (b) Give a couple
> > more
> > > days for folks who tested on the binaries to reverify - I count one
> such
> > > clear vote as of now.
> > >
> > > Thanks
> > > +Vinod
> > >
> > >
> > > On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheeleast@gmail.com<mailto:wh
> > > eeleast@gmail.com>> wrote:
> > >
> > > HI Arpit,
> > >
> > > I think it won't match if we do rebuild. It should be fine as far as
> > > they're signed, correct? I don't see any policy doesn't allow this.
> > >
> > > Thanks,
> > > Wangda
> > >
> > >
> > > On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <
> aagarwal@hortonworks.com<
> > > mailto:aagarwal@hortonworks.com>>
> > > wrote:
> > >
> > > Thanks Wangda, I see the shaded jars now.
> > >
> > > Are the repo jars required to be the same as the binary release? They
> > > don’t match right now, probably they got rebuilt.
> > >
> > > +1 (binding), modulo that remaining question.
> > >
> > > * Verified signatures
> > > * Verified checksums for source and binary artefacts
> > > * Sanity checked jars on r.a.o.
> > > * Built from source
> > > * Deployed to 3 node secure cluster with NameNode HA
> > > * Verified HDFS web UIs
> > > * Tried out HDFS shell commands
> > > * Ran sample MapReduce jobs
> > >
> > > Thanks!
> > >
> > >
> > > ----------------------------------------------------------------------
> > > From: Wangda Tan <wheeleast@gmail.com<mailto:wheeleast@gmail.com>>
> > > Date: Monday, April 2, 2018 at 9:25 PM
> > > To: Arpit Agarwal <aagarwal@hortonworks.com<mailto:
> aagarwal@hortonworks.
> > > com>>
> > > Cc: Gera Shegalov <gera05@gmail.com<mailto:gera05@gmail.com>>,
Sunil
> G <
> > > sunilg@apache.org<mailto:sunilg@apache.org>>, "
> > > yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>" <
> > > yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>>,
> > Hdfs-dev <
> > > hdfs-dev@hadoop.apache.org<mailto:hdfs-dev@hadoop.apache.org>>, Hadoop
> > > Common <common-dev@hadoop.apache.org<mailto:common-dev@hadoop.
> apache.org
> > > >>,
> > > "mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@
> hadoop.apache.org
> > >"
> > > <mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@
> hadoop.apache.org
> > >>,
> > > Vinod Kumar Vavilapalli <vinodkv@apache.org<mailto:vinodkv@apache.org
> >>
> > > Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> > >
> > > As pointed by Arpit, the previously deployed shared jars are incorrect.
> > > Just redeployed jars and staged. @Arpit, could you please check the
> > updated
> > > Maven repo? https://repository.apache.org/content/repositories/
> > > orgapachehadoop-1092
> > >
> > > Since the jars inside binary tarballs are correct (
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> > > need roll another RC, just update Maven repo should be sufficient.
> > >
> > > Best,
> > > Wangda
> > >
> > >
> > > On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheeleast@gmail.com
> >
> > > wrote:
> > > Hi Arpit,
> > >
> > > Thanks for pointing out this.
> > >
> > > I just removed all .md5 files from artifacts. I found md5 checksums
> still
> > > exist in .mds files and I didn't remove them from .mds file because it
> is
> > > generated by create-release script and Apache guidance is "should not"
> > > instead of "must not". Please let me know if you think they need to be
> > > removed as well.
> > >
> > > - Wangda
> > >
> > >
> > >
> > > On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:
> > > aagarwal@hortonworks.com<mailto:aagarwal@hortonworks.com>> wrote:
> > > Thanks for putting together this RC, Wangda.
> > >
> > > The guidance from Apache is to omit MD5s, specifically:
> > > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> > >
> > > https://www.apache.org/dev/release-distribution#sigs-and-sums
> > >
> > >
> > >
> > >
> > > On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheeleast@gmail.com>
> > > wrote:
> > >
> > > Hi Gera,
> > >
> > > It's my bad, I thought only src/bin tarball is enough.
> > >
> > > I just uploaded all other things under artifact/ to
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1/
> > >
> > > Please let me know if you have any other comments.
> > >
> > > Thanks,
> > > Wangda
> > >
> > >
> > > On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:
> gera05@gmail.com>
> > > wrote:
> > >
> > >
> > > Thanks, Wangda!
> > >
> > > There are many more artifacts in previous votes, e.g., see
> > > http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others
> the
> > > site tarball is missing.
> > >
> > > On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sunilg@apache.org>
> > wrote:
> > >
> > >
> > > Thanks Wangda for initiating the release.
> > >
> > > I tested this RC built from source file.
> > >
> > >
> > >  - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
> > > UI.
> > >  - Below feature sanity is done
> > >     - Application priority
> > >     - Application timeout
> > >     - Intra Queue preemption with priority based
> > >     - DS based affinity tests to verify placement constraints.
> > >  - Tested basic NodeLabel scenarios.
> > >     - Added couple of labels to few of nodes and behavior is coming
> > >     correct.
> > >     - Verified old UI  and new YARN UI for labels.
> > >     - Submitted apps to labelled cluster and it works fine.
> > >     - Also performed few cli commands related to nodelabel.
> > >  - Test basic HA cases and seems correct.
> > >  - Tested new YARN UI . All pages are getting loaded correctly.
> > >
> > >
> > > - Sunil
> > >
> > > On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheeleast@gmail.com
> >
> > > wrote:
> > >
> > >
> > > Hi folks,
> > >
> > > Thanks to the many who helped with this release since Dec 2017 [1].
> > > We've
> > >
> > > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
> > >
> > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1
> > >
> > > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> > > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
> > >
> > > The maven artifacts are available via http://repository.apache.org<h
> > > ttp://repository.apache.org/> at
> > > https://repository.apache.org/content/repositories/
> > > orgapachehadoop-1090/
> > >
> > > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
> > >
> > > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> > > include the first class GPU/FPGA support on YARN, Native services,
> > > Support
> > >
> > > rich placement constraints in YARN, S3-related enhancements, allow HDFS
> > > block replicas to be provided by an external storage system, etc.
> > >
> > > For 3.1.0 RC0 vote discussion, please see [3].
> > >
> > > We’d like to use this as a starting release for 3.1.x [1], depending on
> > > how
> > >
> > > it goes, get it stabilized and potentially use a 3.1.1 in several weeks
> > > as
> > >
> > > the stable release.
> > >
> > > We have done testing with a pseudo cluster:
> > > - Ran distributed job.
> > > - GPU scheduling/isolation.
> > > - Placement constraints (intra-application anti-affinity) by using
> > > distributed shell.
> > >
> > > My +1 to start.
> > >
> > > Best,
> > > Wangda/Vinod
> > >
> > > [1]
> > >
> > > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> > > c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<
> > > http://3cyarn-dev.hadoop.apache.org/>
> > > <http://3cyarn-dev.hadoop.apache.org/>%3E
> > >
> > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.0)
> > > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
> > > BY
> > >
> > > fixVersion ASC
> > > [3]
> > >
> > > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> > > 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<
> > > http://3cyarn-dev.hadoop.apache.org/>
> > > <http://3cyarn-dev.hadoop.apache.org/>%3E
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message