hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Kellerman <...@powerset.com>
Subject Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)
Date Mon, 31 Mar 2008 16:10:31 GMT
It appears that the classpath was incorrect during the test runs where hadoop-0.16.2 had failures.

With the classpath corrected, hadoop-0.16.2 passed the HBase stress test 5 times out of 5.

+1 for hadoop-0.16.2

-------------
About the HBase stress test:

The HBase stress test puts both Hadoop and HBase under heavy load, which is why we use it
to qualify HBase releases.

In this test we run a 4 node cluster. All 4 nodes are data nodes.
Node xxx-xxx-12.u.powerset.com runs the name node, JobTracker and
HBase master. The test is also launched from this node. All 4 nodes
run task trackers.

The sole region server runs on xxx-xxx-13.u.powerset.com

The application (PerformanceEvaluation) is launched from xxx-xxx-12.u.powerset.com but runs
map jobs on all 4 nodes of the cluster.

We install everything on -12 and use rsync to make sure all the cluster
nodes are doing the same thing.

Attachments:
env.sh sets up the directories we want to rsync/run from.
rsync.sh is how we push the software around the cluster.
start.sh is how we start up hadoop and hbase.
The command line for running PerformanceEvaluation is:

$ date | tee start.time ; hbase-0.1.0/bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation
sequentialWrite 4 ; date | tee end.time

hadoop-env.sh - how we point hadoop at hbase
hadoop-site.xml - just changes port numbers and hadoop.tmp.dir
masters - points to xxx-xxx-12.u.powerset.com
slaves - points to xxx-xxx-1[2-5].u.powerset.com

For HBase:
hbase-site.xml changes port numbers
regionservers - points to xxx-xxx-13.u.powerset.com
hbase-env.sh - points to JAVA_HOME and HBASE-HOME (only modifications from distribution)

The key seems to be in $HBASE_HOME/bin/hbase
- This file includes all jars in $HBASE_HOME/lib - because that directory contains hadoop
jars, they get put on the classpath which would explain the strange behavior and mis-matched
line numbers. hbase-0.1.0 ships with jars from hadoop-0.16.0. Replacing the 0.16.0 jars in
HBASE_HOME/lib with the jars from hadoop-0.16.2 seems to have solved the problem.

My bad.
---
Jim Kellerman, Senior Engineer; Powerset

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1352 - Release Date: 3/31/2008 10:13 AM



Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message