hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: [VOTE] Release Apache Hadoop 2.1.0-beta
Date Thu, 22 Aug 2013 20:39:46 GMT
+1, binding

Review process

# symlink /usr/local/bin/protoc to the homebrew installed 2.5.0 version

# delete all 2.1.0-beta artifacts in the mvn repo:

  find ~/.m2 -name 2.1.0-beta -print | xargs rm -rf

# checkout hbase apache/branch-0.95 (commit # b58d596 )

# switch to ASF repo (arun's private repo is in the POM, with JARs with the
same sha1 sum, I'm just being rigorous)
        <repository>
          <id>ASF Staging</id>
          <url>https://repository.apache.org/content/groups/staging/</url>
        </repository>

# clean build of hbase tar against the beta artifacts

mvn clean install assembly:single -DskipTests -Dmaven.javadoc.skip=true
-Dhadoop.profile=2.0 -Dhadoop-two.version=2.1.0-beta

# Observe DL taking place

]
[INFO] --- maven-assembly-plugin:2.4:single (default-cli) @ hbase ---
[INFO] Assemblies have been skipped per configuration of the skipAssembly
parameter.
[INFO]

[INFO]
------------------------------------------------------------------------
[INFO] Building HBase - Common 0.95.3-SNAPSHOT
[INFO]
------------------------------------------------------------------------
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.pom(2
KB at 3.3 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project/2.1.0-beta/hadoop-project-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project/2.1.0-beta/hadoop-project-2.1.0-beta.pom(34
KB at 383.8 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-main/2.1.0-beta/hadoop-main-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-main/2.1.0-beta/hadoop-main-2.1.0-beta.pom(17
KB at 184.0 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.pom(26
KB at 293.3 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project-dist/2.1.0-beta/hadoop-project-dist-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project-dist/2.1.0-beta/hadoop-project-dist-2.1.0-beta.pom(17
KB at 176.2 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.pom
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.pom(7
KB at 79.0 KB/sec)
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.jar
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar
Downloading:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.jar
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.jar(17
KB at 146.1 KB/sec)
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.jar(47
KB at 241.7 KB/sec)
Downloaded:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar(2657
KB at 2260.9 KB/sec)
[INFO]


# get md5 sum of hadoop-common-2.1.0-beta artifact in
https://repository.apache.org/content/groups/staging/ :
0166f5c94d3699b3a37efc16ebb1ceea3acb3b53

# verify version of artifact in local m2 repo
    $ sha1sum
~/.m2/repository/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar
    0166f5c94d3699b3a37efc16ebb1ceea3acb3b53

# in hbase/hbase-assembly/target , gunzip then untar the
hbase-0.95.3-SNAPSHOT-bin.tar file

# Patch the Hoya POM to use 2.1.0-beta instead of a local 2.1.1-SNAPSHOT

# run some of the hbase cluster deploy & flexing tests


 mvn clean test  -Pstaging

 (all tests pass after 20 min)

== functional tests =

# build and the Hoya JAR with classpath

mvn package -Pstaging

# D/L the binary .tar.gz file, and scp to an ubuntu VM with the hadoop conf
properties for net-accessible HDFS & YARN services, no memory limits on
containers

https://github.com/hortonworks/hoya/tree/master/src/test/configs/ubuntu
https://github.com/hortonworks/hoya/blob/master/src/test/configs/ubuntu/core-site.xml
https://github.com/hortonworks/hoya/blob/master/src/test/configs/ubuntu/yarn-site.xml

# stop the running hadoop-2.1.1-snapshot cluster

# start the new cluster services

    hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode
    hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode

    yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager
    yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager

# Verify in dfshealth that NN is up, version is built ( 2013-08-15T20:48Z
by hortonmu from branch-2.1.0-beta ). (the NN came up in safe mode as a
couple of in-flight blocks were missing; NN recovered from this happily
after 20s)

# copy hbase-0.93 tarball to HDFS
  hdfs dfs -copyFromLocal hbase-0.95.3-SNAPSHOT-bin.tar
hdfs://ubuntu:9000/hbase.tar

# unfreeze an hbase cluster that was running on hbase-0.95.2 & Hadoop-2.1.1
(protobuf 2.4) versions & set up to use hdfs://ubuntu:9000/hbase.tar as the
hbase image to install

  java -jar target/hoya-0.3-SNAPSHOT.jar org.apache.hadoop.hoya.Hoya thaw
cl1  --manager ubuntu:8032 --filesystem hdfs://ubuntu:9000

# verify cluster is running according to Hoya CLI and YARN web GUI

# point browser at HDFS master: http://ubuntu:8080/master-status , verify
it is happy

# fault injection: ssh-in and kill -9 all HRegionServers

# verify that within 60s the # no. of region servers is as desired; YARN
notified Hoya of container loss; replacement containers requested and
region servers deployed.

# Freeze the cluster:
    java -jar target/hoya-0.3-SNAPSHOT.jar org.apache.hadoop.hoya.Hoya\
    freeze cl1 \
    --manager ubuntu:8032 --filesystem hdfs://ubuntu:9000

# verify that hoya list command and GUI show app as finished; HBase cluster
is no longer present

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message