Return-Path: Delivered-To: apmail-hadoop-hbase-commits-archive@minotaur.apache.org Received: (qmail 71155 invoked from network); 31 Dec 2009 10:51:19 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 31 Dec 2009 10:51:19 -0000 Received: (qmail 50902 invoked by uid 500); 31 Dec 2009 10:51:19 -0000 Delivered-To: apmail-hadoop-hbase-commits-archive@hadoop.apache.org Received: (qmail 50868 invoked by uid 500); 31 Dec 2009 10:51:19 -0000 Mailing-List: contact hbase-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hbase-dev@hadoop.apache.org Delivered-To: mailing list hbase-commits@hadoop.apache.org Received: (qmail 50859 invoked by uid 99); 31 Dec 2009 10:51:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Dec 2009 10:51:19 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Dec 2009 10:51:12 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 5F6A423889D2; Thu, 31 Dec 2009 10:50:51 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r894808 - in /hadoop/hbase/branches/0.20: CHANGES.txt src/contrib/ec2/bin/hbase-ec2-init-remote.sh Date: Thu, 31 Dec 2009 10:50:51 -0000 To: hbase-commits@hadoop.apache.org From: apurtell@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20091231105051.5F6A423889D2@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: apurtell Date: Thu Dec 31 10:50:50 2009 New Revision: 894808 URL: http://svn.apache.org/viewvc?rev=894808&view=rev Log: HBASE-2080 [EC2] Support multivolume local instance storage Modified: hadoop/hbase/branches/0.20/CHANGES.txt hadoop/hbase/branches/0.20/src/contrib/ec2/bin/hbase-ec2-init-remote.sh Modified: hadoop/hbase/branches/0.20/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/hbase/branches/0.20/CHANGES.txt?rev=894808&r1=894807&r2=894808&view=diff ============================================================================== --- hadoop/hbase/branches/0.20/CHANGES.txt (original) +++ hadoop/hbase/branches/0.20/CHANGES.txt Thu Dec 31 10:50:50 2009 @@ -36,6 +36,7 @@ (Lars George via JD) HBASE-2082 TableInputFormat is ignoring input scan's stop row setting (Scott Wang via Andrew Purtell) + HBASE-2080 [EC2] Support multivolume local instance storage IMPROVEMENTS HBASE-1961 HBase EC2 scripts Modified: hadoop/hbase/branches/0.20/src/contrib/ec2/bin/hbase-ec2-init-remote.sh URL: http://svn.apache.org/viewvc/hadoop/hbase/branches/0.20/src/contrib/ec2/bin/hbase-ec2-init-remote.sh?rev=894808&r1=894807&r2=894808&view=diff ============================================================================== --- hadoop/hbase/branches/0.20/src/contrib/ec2/bin/hbase-ec2-init-remote.sh (original) +++ hadoop/hbase/branches/0.20/src/contrib/ec2/bin/hbase-ec2-init-remote.sh Thu Dec 31 10:50:50 2009 @@ -16,6 +16,68 @@ HBASE_HOME=`ls -d /usr/local/hbase-*` HBASE_VERSION=`echo $HBASE_HOME | cut -d '-' -f 2` +export USER="root" + +# up open file descriptor limits +echo "root soft nofile 32768" >> /etc/security/limits.conf +echo "root hard nofile 32768" >> /etc/security/limits.conf + +# up epoll limits; ok if this fails, only valid for kernels 2.6.27+ +sysctl -w fs.epoll.max_user_instances=32768 > /dev/null 2>&1 + +[ ! -f /etc/hosts ] && echo "127.0.0.1 localhost" > /etc/hosts + +# Extra packages + +if [ "$EXTRA_PACKAGES" != "" ] ; then + # format should be ... + pkg=( $EXTRA_PACKAGES ) + wget -nv -O /etc/yum.repos.d/user.repo ${pkg[0]} + yum -y update yum + yum -y install ${pkg[@]:1} +fi + +# Ganglia + +if [ "$IS_MASTER" = "true" ]; then + sed -i -e "s|\( *mcast_join *=.*\)|#\1|" \ + -e "s|\( *bind *=.*\)|#\1|" \ + -e "s|\( *mute *=.*\)| mute = yes|" \ + -e "s|\( *location *=.*\)| location = \"master-node\"|" \ + /etc/gmond.conf + mkdir -p /mnt/ganglia/rrds + chown -R ganglia:ganglia /mnt/ganglia/rrds + rm -rf /var/lib/ganglia; cd /var/lib; ln -s /mnt/ganglia ganglia; cd + service gmond start + service gmetad start + apachectl start +else + sed -i -e "s|\( *mcast_join *=.*\)|#\1|" \ + -e "s|\( *bind *=.*\)|#\1|" \ + -e "s|\(udp_send_channel {\)|\1\n host=$MASTER_HOST|" \ + /etc/gmond.conf + service gmond start +fi + +# Probe for instance volumes + +# /dev/sdb as /mnt is always set up by base image +DFS_NAME_DIR="/mnt/hadoop/dfs/name" +DFS_DATA_DIR="/mnt/hadoop/dfs/data" +i=2 +for d in c d e f g h i j k l m n o p q r s t u v w x y z; do + m="/mnt${i}" + mkdir -p $m + mount /dev/sd${d} $m > /dev/null 2>&1 + if [ $? -eq 0 ] ; then + if [ $i -lt 3 ] ; then # no more than two namedirs + DFS_NAME_DIR="${DFS_NAME_DIR},${m}/hadoop/dfs/name" + fi + DFS_DATA_DIR="${DFS_DATA_DIR},${m}/hadoop/dfs/data" + i=$(( i + 1 )) + fi +done + # Hadoop configuration cat > $HADOOP_HOME/conf/core-site.xml < EOF - cat > $HADOOP_HOME/conf/hdfs-site.xml < @@ -41,9 +102,16 @@ fs.default.name hdfs://$MASTER_HOST:8020 + + dfs.name.dir + $DFS_NAME_DIR + + + dfs.data.dir + $DFS_DATA_DIR + EOF - cat > $HADOOP_HOME/conf/mapred-site.xml < @@ -66,12 +134,10 @@ EOF - # Update classpath to include HBase jars and config cat >> $HADOOP_HOME/conf/hadoop-env.sh < $HADOOP_HOME/conf/hadoop-metrics.properties < EOF - # Override JVM options cat >> $HBASE_HOME/conf/hbase-env.sh < $HBASE_HOME/conf/hadoop-metrics.properties <> /etc/security/limits.conf -echo "root hard nofile 32768" >> /etc/security/limits.conf - -# up epoll limits; ok if this fails, only valid for kernels 2.6.27+ -sysctl -w fs.epoll.max_user_instances=32768 > /dev/null 2>&1 - mkdir -p /mnt/hadoop/logs /mnt/hbase/logs -[ ! -f /etc/hosts ] && echo "127.0.0.1 localhost" > /etc/hosts - -export USER="root" - -if [ "$EXTRA_PACKAGES" != "" ] ; then - # format should be ... - # this will only work with bash - pkg=( $EXTRA_PACKAGES ) - wget -nv -O /etc/yum.repos.d/user.repo ${pkg[0]} - yum -y update yum - yum -y install ${pkg[@]:1} -fi - if [ "$IS_MASTER" = "true" ]; then - # MASTER - # Prep Ganglia - sed -i -e "s|\( *mcast_join *=.*\)|#\1|" \ - -e "s|\( *bind *=.*\)|#\1|" \ - -e "s|\( *mute *=.*\)| mute = yes|" \ - -e "s|\( *location *=.*\)| location = \"master-node\"|" \ - /etc/gmond.conf - mkdir -p /mnt/ganglia/rrds - chown -R ganglia:ganglia /mnt/ganglia/rrds - rm -rf /var/lib/ganglia; cd /var/lib; ln -s /mnt/ganglia ganglia; cd - service gmond start - service gmetad start - apachectl start - # only format on first boot - [ ! -e /mnt/hadoop/dfs ] && "$HADOOP_HOME"/bin/hadoop namenode -format - + [ ! -e /mnt/hadoop/dfs/name ] && "$HADOOP_HOME"/bin/hadoop namenode -format "$HADOOP_HOME"/bin/hadoop-daemon.sh start namenode "$HADOOP_HOME"/bin/hadoop-daemon.sh start datanode "$HADOOP_HOME"/bin/hadoop-daemon.sh start jobtracker - sleep 10 "$HBASE_HOME"/bin/hbase-daemon.sh start master - else - - # SLAVE - - # Prep Ganglia - sed -i -e "s|\( *mcast_join *=.*\)|#\1|" \ - -e "s|\( *bind *=.*\)|#\1|" \ - -e "s|\(udp_send_channel {\)|\1\n host=$MASTER_HOST|" \ - /etc/gmond.conf - service gmond start "$HADOOP_HOME"/bin/hadoop-daemon.sh start datanode "$HBASE_HOME"/bin/hbase-daemon.sh start regionserver "$HADOOP_HOME"/bin/hadoop-daemon.sh start tasktracker - fi -# Run this script on next boot rm -f /var/ec2/ec2-run-user-data.*