Return-Path: X-Original-To: apmail-incubator-hama-commits-archive@minotaur.apache.org Delivered-To: apmail-incubator-hama-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 70E5A9F2A for ; Tue, 27 Sep 2011 09:36:37 +0000 (UTC) Received: (qmail 58541 invoked by uid 500); 27 Sep 2011 09:36:37 -0000 Delivered-To: apmail-incubator-hama-commits-archive@incubator.apache.org Received: (qmail 58512 invoked by uid 500); 27 Sep 2011 09:36:37 -0000 Mailing-List: contact hama-commits-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hama-dev@incubator.apache.org Delivered-To: mailing list hama-commits@incubator.apache.org Received: (qmail 58488 invoked by uid 99); 27 Sep 2011 09:36:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Sep 2011 09:36:37 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,WEIRD_QUOTING X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Sep 2011 09:36:25 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 1F7212388ABA; Tue, 27 Sep 2011 09:36:02 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1176297 [4/19] - in /incubator/hama/branches/HamaV2: ./ api/ api/target/ api/target/classes/ api/target/classes/META-INF/ api/target/lib/ api/target/maven-archiver/ api/target/maven-shared-archive-resources/ api/target/maven-shared-archive... Date: Tue, 27 Sep 2011 09:35:48 -0000 To: hama-commits@incubator.apache.org From: tjungblut@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20110927093602.1F7212388ABA@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Added: incubator/hama/branches/HamaV2/client/target/test-classes/META-INF/DEPENDENCIES URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/client/target/test-classes/META-INF/DEPENDENCIES?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/client/target/test-classes/META-INF/DEPENDENCIES (added) +++ incubator/hama/branches/HamaV2/client/target/test-classes/META-INF/DEPENDENCIES Tue Sep 27 09:35:21 2011 @@ -0,0 +1,103 @@ +// ------------------------------------------------------------------ +// Transitive dependencies of this project determined from the +// maven pom organized by organization. +// ------------------------------------------------------------------ + +Apache Hama Client + + +From: 'an unknown organization' + - geronimo-spec-jta geronimo-spec:geronimo-spec-jta:jar:1.0.1B-rc4 + + - HSQLDB (http://hsqldb.org/) hsqldb:hsqldb:jar:1.8.0.10 + License: HSQLDB License (http://hsqldb.org/web/hsqlLicense.html) + - JLine (http://jline.sourceforge.net) jline:jline:jar:0.9.94 + License: BSD (LICENSE.txt) + - An open source Java toolkit for Amazon S3 (http://jets3t.s3.amazonaws.com/index.html) net.java.dev.jets3t:jets3t:jar:0.7.1 + License: Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0) + - kosmosfs (http://kosmosfs.sourceforge.net/) net.sf.kosmosfs:kfs:jar:0.3 + License: The Apache Software License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt) + - hadoop-core org.apache.hadoop:hadoop-core:jar:0.20.2 + + - hadoop-test org.apache.hadoop:hadoop-test:jar:0.20.2 + + - servlet-api org.apache.tomcat:servlet-api:jar:6.0.32 + + - zookeeper org.apache.zookeeper:zookeeper:jar:3.3.1 + + - Eclipse JDT Core (http://www.eclipse.org/jdt/) org.eclipse.jdt:core:jar:3.1.1 + License: Eclipse Public License v1.0 (http://www.eclipse.org/org/documents/epl-v10.php) + - oro oro:oro:jar:2.0.8 + + - jasper-compiler tomcat:jasper-compiler:jar:5.5.12 + + - jasper-runtime tomcat:jasper-runtime:jar:5.5.12 + + - xmlenc Library (http://xmlenc.sourceforge.net) xmlenc:xmlenc:jar:0.52 + License: The BSD License (http://www.opensource.org/licenses/bsd-license.php) + +From: 'Apache MINA Project' (http://mina.apache.org/) + - Apache MINA Core (http://mina.apache.org/mina-core) org.apache.mina:mina-core:bundle:2.0.0-M5 + License: Apache 2.0 License (http://www.apache.org/licenses/LICENSE-2.0) + +From: 'Apache Software Foundation' (http://jakarta.apache.org/) + - HttpClient (http://jakarta.apache.org/commons/httpclient/) commons-httpclient:commons-httpclient:jar:3.0.1 + License: Apache License (http://www.apache.org/licenses/LICENSE-2.0) + +From: 'Apache Software Foundation' (http://www.apache.org) + - Annotation 1.0 (http://geronimo.apache.org/specs/geronimo-annotation_1.0_spec) org.apache.geronimo.specs:geronimo-annotation_1.0_spec:jar:1.0 + License: The Apache Software License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt) + +From: 'Apache Software Foundation' + - org.apache.tools.ant (http://ant.apache.org/ant/) org.apache.ant:ant:jar:1.7.1 + + - ant-launcher (http://ant.apache.org/ant-launcher/) org.apache.ant:ant-launcher:jar:1.7.1 + + +From: 'Mort Bay Consulting' (http://www.mortbay.com) + - Jetty Server (http://jetty.mortbay.org/project/modules/jetty) org.mortbay.jetty:jetty:jar:6.1.14 + License: Apache License Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0) + - Servlet Annotations (http://jetty.mortbay.org/project/jetty-annotations) org.mortbay.jetty:jetty-annotations:jar:6.1.14 + License: Apache License Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0) + - Jetty Plus (http://jetty.mortbay.org/project/jetty-plus) org.mortbay.jetty:jetty-plus:jar:6.1.14 + License: Apache License Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0) + - Jetty Utilities (http://jetty.mortbay.org/project/jetty-util) org.mortbay.jetty:jetty-util:jar:6.1.14 + License: Apache License Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0) + - Glassfish Jasper (http://jetty.mortbay.org/project/modules/jsp-2.1) org.mortbay.jetty:jsp-2.1:jar:6.1.14 + License: CDDL 1.0 (https://glassfish.dev.java.net/public/CDDLv1.0.html) + - Glassfish Jasper API (http://jetty.mortbay.org/project/modules/jsp-api-2.1) org.mortbay.jetty:jsp-api-2.1:jar:6.1.14 + License: Apache License Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0) + - Servlet Specification 2.5 API (http://jetty.mortbay.org/project/modules/servlet-api-2.5) org.mortbay.jetty:servlet-api-2.5:jar:6.1.14 + License: CDDL 1.0 (https://glassfish.dev.java.net/public/CDDLv1.0.html) + +From: 'QOS.ch' (http://www.qos.ch) + - SLF4J API Module (http://www.slf4j.org) org.slf4j:slf4j-api:jar:1.5.2 + + - SLF4J LOG4J-12 Binding (http://www.slf4j.org) org.slf4j:slf4j-log4j12:jar:1.5.2 + + +From: 'The Apache Software Foundation' (http://jakarta.apache.org) + - Codec (http://jakarta.apache.org/commons/codec/) commons-codec:commons-codec:jar:1.3 + License: The Apache Software License, Version 2.0 (/LICENSE.txt) + - EL (http://jakarta.apache.org/commons/el/) commons-el:commons-el:jar:1.0 + License: The Apache Software License, Version 2.0 (/LICENSE.txt) + - Jakarta Commons Net (http://jakarta.apache.org/commons/${pom.artifactId.substring(8)}/) commons-net:commons-net:jar:1.4.1 + License: The Apache Software License, Version 2.0 (/LICENSE.txt) + +From: 'The Apache Software Foundation' (http://www.apache.org/) + - ant (http://www.apache.org/ant/) ant:ant:jar:1.6.5 + License: The Apache Software License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt) + - Commons CLI (http://commons.apache.org/cli/) commons-cli:commons-cli:jar:1.2 + License: The Apache Software License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt) + - Commons Logging (http://commons.apache.org/logging) commons-logging:commons-logging:jar:1.1.1 + License: The Apache Software License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt) + - Apache Ftplet API (http://mina.apache.org/ftpserver) org.apache.ftpserver:ftplet-api:bundle:1.0.0 + License: Apache 2.0 License (http://www.apache.org/licenses/LICENSE-2.0) + - Apache FtpServer Core (http://mina.apache.org/ftpserver/ftpserver-core) org.apache.ftpserver:ftpserver-core:bundle:1.0.0 + License: Apache 2.0 License (http://www.apache.org/licenses/LICENSE-2.0) + - Apache FtpServer Deprecated classes (http://mina.apache.org/ftpserver/ftpserver-deprecated) org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2 + License: Apache 2.0 License (http://www.apache.org/licenses/LICENSE-2.0) + + + + Added: incubator/hama/branches/HamaV2/client/target/test-classes/META-INF/LICENSE URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/client/target/test-classes/META-INF/LICENSE?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/client/target/test-classes/META-INF/LICENSE (added) +++ incubator/hama/branches/HamaV2/client/target/test-classes/META-INF/LICENSE Tue Sep 27 09:35:21 2011 @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. Propchange: incubator/hama/branches/HamaV2/client/target/test-classes/META-INF/LICENSE ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/client/target/test-classes/META-INF/NOTICE URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/client/target/test-classes/META-INF/NOTICE?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/client/target/test-classes/META-INF/NOTICE (added) +++ incubator/hama/branches/HamaV2/client/target/test-classes/META-INF/NOTICE Tue Sep 27 09:35:21 2011 @@ -0,0 +1,8 @@ + +Apache Hama Client +Copyright 2008-2011 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + Propchange: incubator/hama/branches/HamaV2/client/target/test-classes/META-INF/NOTICE ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/bin/grooms.sh URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/bin/grooms.sh?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/bin/grooms.sh (added) +++ incubator/hama/branches/HamaV2/core/bin/grooms.sh Tue Sep 27 09:35:21 2011 @@ -0,0 +1,68 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# Run a shell command on all slave hosts. +# +# Environment Variables +# +# HAMA_GROOMS File naming remote hosts. +# Default is ${HAMA_CONF_DIR}/groomservers. +# HAMA_CONF_DIR Alternate conf dir. Default is ${HAMA_HOME}/conf. +# HAMA_GROOM_SLEEP Seconds to sleep between spawning remote commands. +# HAMA_SSH_OPTS Options passed to ssh when running remote commands. +## + +usage="Usage: grooms.sh [--config confdir] command..." + +# if no args specified, show usage +if [ $# -le 0 ]; then + echo $usage + exit 1 +fi + +bin=`dirname "$0"` +bin=`cd "$bin"; pwd` + +. "$bin"/hama-config.sh + +# If the groomservers file is specified in the command line, +# then it takes precedence over the definition in +# hama-env.sh. Save it here. +HOSTLIST=$HAMA_GROOMS + +if [ -f "${HAMA_CONF_DIR}/hama-env.sh" ]; then + . "${HAMA_CONF_DIR}/hama-env.sh" +fi + +if [ "$HOSTLIST" = "" ]; then + if [ "$HAMA_GROOMS" = "" ]; then + export HOSTLIST="${HAMA_CONF_DIR}/groomservers" + else + export HOSTLIST="${HAMA_GROOMS}" + fi +fi + +for groom in `cat "$HOSTLIST"|sed "s/#.*$//;/^$/d"`; do + ssh $HAMA_SSH_OPTS $groom $"${@// /\\ }" \ + 2>&1 | sed "s/^/$groom: /" & + if [ "$HAMA_GROOM_SLEEP" != "" ]; then + sleep $HAMA_GROOM_SLEEP + fi +done + +wait Propchange: incubator/hama/branches/HamaV2/core/bin/grooms.sh ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/bin/hama URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/bin/hama?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/bin/hama (added) +++ incubator/hama/branches/HamaV2/core/bin/hama Tue Sep 27 09:35:21 2011 @@ -0,0 +1,186 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# The Hama command script +# +# Environment Variables +# +# JAVA_HOME The java implementation to use. Overrides JAVA_HOME. +# +# HAMA_CLASSPATH Extra Java CLASSPATH entries. +# +# HAMA_HEAPSIZE The maximum amount of heap to use, in MB. +# Default is 1000. +# +# HAMA_OPTS Extra Java runtime options. +# +# HAMA_NAMENODE_OPTS These options are added to HAMA_OPTS +# HAMA_CLIENT_OPTS when the respective command is run. +# HAMA_{COMMAND}_OPTS etc HAMA_JT_OPTS applies to JobTracker +# for e.g. HAMA_CLIENT_OPTS applies to +# more than one command (fs, dfs, fsck, +# dfsadmin etc) +# +# HAMA_CONF_DIR Alternate conf dir. Default is ${HAMA_HOME}/conf. +# +# HAMA_ROOT_LOGGER The root appender. Default is INFO,console +# + +bin=`dirname "$0"` +bin=`cd "$bin"; pwd` + +. "$bin"/hama-config.sh + +cygwin=false +case "`uname`" in +CYGWIN*) cygwin=true;; +esac + +# if no args specified, show usage +if [ $# = 0 ]; then + echo "Usage: hama [--config confdir] COMMAND" + echo "where COMMAND is one of:" + echo " bspmaster run the BSP Master node" + echo " groom run the Groom node" + echo " zookeeper run a Zookeeper server" + echo " job manipulate BSP jobs" + echo " jar run a jar file" + echo " or" + echo " CLASSNAME run the class named CLASSNAME" + echo "Most commands print help when invoked w/o parameters." + exit 1 +fi + +# get arguments +COMMAND=$1 +shift + +if [ -f "${HAMA_CONF_DIR}/hama-env.sh" ]; then + . "${HAMA_CONF_DIR}/hama-env.sh" +fi + +# some Java parameters +if [ "$JAVA_HOME" != "" ]; then + #echo "run java in $JAVA_HOME" + JAVA_HOME=$JAVA_HOME +fi + +if [ "$JAVA_HOME" = "" ]; then + echo "Error: JAVA_HOME is not set." + exit 1 +fi + +JAVA=$JAVA_HOME/bin/java +JAVA_HEAP_MAX=-Xmx1000m + +# check envvars which might override default args +if [ "$HAMA_HEAPSIZE" != "" ]; then + #echo "run with heapsize $HAMA_HEAPSIZE" + JAVA_HEAP_MAX="-Xmx""$HAMA_HEAPSIZE""m" + #echo $JAVA_HEAP_MAX +fi + +# CLASSPATH initially contains $HAMA_CONF_DIR +CLASSPATH="${HAMA_CONF_DIR}" +CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar + +# for developers, add Hama classes to CLASSPATH +if [ -d "$HAMA_HOME/target/classes" ]; then + CLASSPATH=${CLASSPATH}:$HAMA_HOME/target/classes +fi +if [ -d "$HAMA_HOME/target/test-classes/classes" ]; then + CLASSPATH=${CLASSPATH}:$HAMA_HOME/target/test-classes +fi + +# so that filenames w/ spaces are handled correctly in loops below +IFS= + +# for releases, add core hama jar to CLASSPATH +for f in $HAMA_HOME/hama-**.jar; do + CLASSPATH=${CLASSPATH}:$f; +done + +# add libs to CLASSPATH +for f in $HAMA_HOME/lib/*.jar; do + CLASSPATH=${CLASSPATH}:$f; +done + +# add user-specified CLASSPATH last +if [ "$HAMA_CLASSPATH" != "" ]; then + CLASSPATH=${CLASSPATH}:${HAMA_CLASSPATH} +fi + +# default log directory & file +if [ "$HAMA_LOG_DIR" = "" ]; then + HAMA_LOG_DIR="$HAMA_HOME/logs" +fi +if [ "$HAMA_LOGFILE" = "" ]; then + HAMA_LOGFILE='hama.log' +fi + +# default policy file for service-level authorization +if [ "$HAMA_POLICYFILE" = "" ]; then + HAMA_POLICYFILE="hama-policy.xml" +fi + +# restore ordinary behaviour +unset IFS + +# figure out which class to run +if [ "$COMMAND" = "bspmaster" ] ; then + CLASS='org.apache.hama.BSPMasterRunner' + BSP_OPTS="$BSP_OPTS $BSP_BSPMASTER_OPTS" +elif [ "$COMMAND" = "groom" ] ; then + CLASS='org.apache.hama.GroomServerRunner' + BSP_OPTS="$BSP_OPTS $BSP_GROOMSERVER_OPTS" +elif [ "$COMMAND" = "zookeeper" ] ; then + CLASS='org.apache.hama.ZooKeeperRunner' +elif [ "$COMMAND" = "job" ] ; then + CLASS='org.apache.hama.bsp.BSPJobClient' +elif [ "$COMMAND" = "jar" ] ; then + CLASS=org.apache.hama.util.RunJar + BSP_OPTS="$BSP_OPTS" +else + CLASS=$COMMAND +fi + +# cygwin path translation +if $cygwin; then + CLASSPATH=`cygpath -p -w "$CLASSPATH"` + HAMA_HOME=`cygpath -w "$HAMA_HOME"` + HAMA_LOG_DIR=`cygpath -w "$HAMA_LOG_DIR"` + TOOL_PATH=`cygpath -p -w "$TOOL_PATH"` +fi + +# cygwin path translation +if $cygwin; then + JAVA_LIBRARY_PATH=`cygpath -p "$JAVA_LIBRARY_PATH"` +fi + +HAMA_OPTS="$HAMA_OPTS -Dhama.log.dir=$HAMA_LOG_DIR" +HAMA_OPTS="$HAMA_OPTS -Dhama.log.file=$HAMA_LOGFILE" +HAMA_OPTS="$HAMA_OPTS -Dhama.home.dir=$HAMA_HOME" +HAMA_OPTS="$HAMA_OPTS -Dhama.id.str=$HAMA_IDENT_STRING" +HAMA_OPTS="$HAMA_OPTS -Dhama.root.logger=${HAMA_ROOT_LOGGER:-INFO,console}" +if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then + HAMA_OPTS="$HAMA_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH" +fi +HAMA_OPTS="$HAMA_OPTS -Dhama.policy.file=$HAMA_POLICYFILE" + +# run it +exec "$JAVA" $JAVA_HEAP_MAX $HAMA_OPTS -classpath "$CLASSPATH" $CLASS "$@" Added: incubator/hama/branches/HamaV2/core/bin/hama-config.sh URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/bin/hama-config.sh?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/bin/hama-config.sh (added) +++ incubator/hama/branches/HamaV2/core/bin/hama-config.sh Tue Sep 27 09:35:21 2011 @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# included in all the hama scripts with source command +# should not be executable directly +# also should not be passed any arguments, since we need original $* + +# resolve links - $0 may be a softlink + +this="$0" +while [ -h "$this" ]; do + ls=`ls -ld "$this"` + link=`expr "$ls" : '.*-> \(.*\)$'` + if expr "$link" : '.*/.*' > /dev/null; then + this="$link" + else + this=`dirname "$this"`/"$link" + fi +done + +# convert relative path to absolute path +bin=`dirname "$this"` +script=`basename "$this"` +bin=`cd "$bin"; pwd` +this="$bin/$script" + +# the root of the Hama installation +export HAMA_HOME=`dirname "$this"`/.. + +#check to see if the conf dir is given as an optional argument +if [ $# -gt 1 ] +then + if [ "--config" = "$1" ] + then + shift + confdir=$1 + shift + HAMA_CONF_DIR=$confdir + fi +fi + +# Allow alternate conf dir location. +HAMA_CONF_DIR="${HAMA_CONF_DIR:-$HAMA_HOME/conf}" + +# Source the hama-env.sh. +# Will have JAVA_HOME and HAMA_MANAGES_ZK defined. +if [ -f "${HAMA_CONF_DIR}/hama-env.sh" ]; then + . "${HAMA_CONF_DIR}/hama-env.sh" +fi + +#check to see it is specified whether to use the grooms or the +# masters file +if [ $# -gt 1 ] +then + if [ "--hosts" = "$1" ] + then + shift + grooms=$1 + shift + export HAMA_SLAVES="${HAMA_CONF_DIR}/$grooms" + fi +fi Propchange: incubator/hama/branches/HamaV2/core/bin/hama-config.sh ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/bin/hama-daemon.sh URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/bin/hama-daemon.sh?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/bin/hama-daemon.sh (added) +++ incubator/hama/branches/HamaV2/core/bin/hama-daemon.sh Tue Sep 27 09:35:21 2011 @@ -0,0 +1,141 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# Runs a Hama command as a daemon. +# +# Environment Variables +# +# HAMA_CONF_DIR Alternate conf dir. Default is ${HAMA_HOME}/conf. +# HAMA_LOG_DIR Where log files are stored. PWD by default. +# HAMA_MASTER host:path where hama code should be rsync'd from +# HAMA_PID_DIR The pid files are stored. /tmp by default. +# HAMA_IDENT_STRING A string representing this instance of hama. $USER by default +# HAMA_NICENESS The scheduling priority for daemons. Defaults to 0. +## + +usage="Usage: hama-daemon.sh [--config ] [--hosts hostlistfile] (start|stop) " + +# if no args specified, show usage +if [ $# -le 1 ]; then + echo $usage + exit 1 +fi + +bin=`dirname "$0"` +bin=`cd "$bin"; pwd` + +. "$bin"/hama-config.sh + +# get arguments +startStop=$1 +shift +command=$1 +shift + +hama_rotate_log () +{ + log=$1; + num=5; + if [ -n "$2" ]; then + num=$2 + fi + if [ -f "$log" ]; then # rotate logs + while [ $num -gt 1 ]; do + prev=`expr $num - 1` + [ -f "$log.$prev" ] && mv "$log.$prev" "$log.$num" + num=$prev + done + mv "$log" "$log.$num"; + fi +} + +if [ -f "${HAMA_CONF_DIR}/hama-env.sh" ]; then + . "${HAMA_CONF_DIR}/hama-env.sh" +fi + +# get log directory +if [ "$HAMA_LOG_DIR" = "" ]; then + export HAMA_LOG_DIR="$HAMA_HOME/logs" +fi +mkdir -p "$HAMA_LOG_DIR" + +if [ "$HAMA_PID_DIR" = "" ]; then + HAMA_PID_DIR=/tmp +fi + +if [ "$HAMA_IDENT_STRING" = "" ]; then + export HAMA_IDENT_STRING="$USER" +fi + +# some variables +export HAMA_LOGFILE=hama-$HAMA_IDENT_STRING-$command-$HOSTNAME.log +export HAMA_ROOT_LOGGER="INFO,DRFA" +log=$HAMA_LOG_DIR/hama-$HAMA_IDENT_STRING-$command-$HOSTNAME.out +pid=$HAMA_PID_DIR/hama-$HAMA_IDENT_STRING-$command.pid + +# Set default scheduling priority +if [ "$HAMA_NICENESS" = "" ]; then + export HAMA_NICENESS=0 +fi + +case $startStop in + + (start) + + mkdir -p "$HAMA_PID_DIR" + + if [ -f $pid ]; then + if kill -0 `cat $pid` > /dev/null 2>&1; then + echo $command running as process `cat $pid`. Stop it first. + exit 1 + fi + fi + + if [ "$HAMA_MASTER" != "" ]; then + echo rsync from $HAMA_MASTER + rsync -a -e ssh --delete --exclude=.svn --exclude='logs/*' --exclude='contrib/hod/logs/*' $HAMA_MASTER/ "$HAMA_HOME" + fi + + hama_rotate_log $log + echo starting $command, logging to $log + cd "$HAMA_HOME" + nohup nice -n $HAMA_NICENESS "$HAMA_HOME"/bin/hama --config $HAMA_CONF_DIR $command "$@" > "$log" 2>&1 < /dev/null & + echo $! > $pid + sleep 1; head "$log" + ;; + + (stop) + + if [ -f $pid ]; then + if kill -0 `cat $pid` > /dev/null 2>&1; then + echo stopping $command + kill `cat $pid` + else + echo no $command to stop + fi + else + echo no $command to stop + fi + ;; + + (*) + echo $usage + exit 1 + ;; + +esac Propchange: incubator/hama/branches/HamaV2/core/bin/hama-daemon.sh ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/bin/hama-daemons.sh URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/bin/hama-daemons.sh?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/bin/hama-daemons.sh (added) +++ incubator/hama/branches/HamaV2/core/bin/hama-daemons.sh Tue Sep 27 09:35:21 2011 @@ -0,0 +1,45 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# Run a Hama command on all slave hosts. + +usage="Usage: hama-daemons.sh [--config confdir] [--hosts hostlistfile] [start|stop] command args..." + +# if no args specified, show usage +if [ $# -le 1 ]; then + echo $usage + exit 1 +fi + +bin=`dirname "$0"` +bin=`cd "$bin"; pwd` + +. $bin/hama-config.sh + +remote_cmd="cd ${HAMA_HOME}; $bin/hama-daemon.sh --config ${HAMA_CONF_DIR} $@" +args="--config ${HAMA_CONF_DIR} $remote_cmd" +command=$2 + +case $command in + (zookeeper) + exec "$bin/zookeepers.sh" $args + ;; + (*) + exec "$bin/grooms.sh" $args + ;; +esac \ No newline at end of file Propchange: incubator/hama/branches/HamaV2/core/bin/hama-daemons.sh ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/bin/start-bspd.sh URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/bin/start-bspd.sh?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/bin/start-bspd.sh (added) +++ incubator/hama/branches/HamaV2/core/bin/start-bspd.sh Tue Sep 27 09:35:21 2011 @@ -0,0 +1,30 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# Start hama map reduce daemons. Run this on master node. + +bin=`dirname "$0"` +bin=`cd "$bin"; pwd` + +. "$bin"/hama-config.sh + +# start bsp daemons +# start zookeeper first to minimize connection errors at startup +"$bin"/hama-daemons.sh --config "${HAMA_CONF_DIR}" start zookeeper +"$bin"/hama-daemon.sh --config $HAMA_CONF_DIR start bspmaster +"$bin"/hama-daemons.sh --config $HAMA_CONF_DIR start groom Propchange: incubator/hama/branches/HamaV2/core/bin/start-bspd.sh ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/bin/stop-bspd.sh URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/bin/stop-bspd.sh?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/bin/stop-bspd.sh (added) +++ incubator/hama/branches/HamaV2/core/bin/stop-bspd.sh Tue Sep 27 09:35:21 2011 @@ -0,0 +1,28 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# Stop hama map reduce daemons. Run this on master node. + +bin=`dirname "$0"` +bin=`cd "$bin"; pwd` + +. "$bin"/hama-config.sh + +"$bin"/hama-daemon.sh --config $HAMA_CONF_DIR stop bspmaster +"$bin"/hama-daemons.sh --config $HAMA_CONF_DIR stop groom +"$bin"/hama-daemons.sh --config "${HAMA_CONF_DIR}" stop zookeeper Propchange: incubator/hama/branches/HamaV2/core/bin/stop-bspd.sh ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/bin/zookeepers.sh URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/bin/zookeepers.sh?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/bin/zookeepers.sh (added) +++ incubator/hama/branches/HamaV2/core/bin/zookeepers.sh Tue Sep 27 09:35:21 2011 @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# +#/** +# * Copyright 2009 The Apache Software Foundation +# * +# * Licensed to the Apache Software Foundation (ASF) under one +# * or more contributor license agreements. See the NOTICE file +# * distributed with this work for additional information +# * regarding copyright ownership. The ASF licenses this file +# * to you under the Apache License, Version 2.0 (the +# * "License"); you may not use this file except in compliance +# * with the License. You may obtain a copy of the License at +# * +# * http://www.apache.org/licenses/LICENSE-2.0 +# * +# * Unless required by applicable law or agreed to in writing, software +# * distributed under the License is distributed on an "AS IS" BASIS, +# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# * See the License for the specific language governing permissions and +# * limitations under the License. +# */ +# +# Run a shell command on all zookeeper hosts. +# +# Environment Variables +# +# HAMA_CONF_DIR Alternate hama conf dir. Default is ${HAMA_HOME}/conf. +# HAMA_SSH_OPTS Options passed to ssh when running remote commands. +# +# Modelled after $HADOOP_HOME/bin/slaves.sh. + +usage="Usage: zookeepers [--config ] command..." + +# if no args specified, show usage +if [ $# -le 0 ]; then + echo $usage + exit 1 +fi + +bin=`dirname "$0"` +bin=`cd "$bin"; pwd` + +. "$bin"/hama-config.sh + +if [ "$HAMA_MANAGES_ZK" = "" ]; then + HAMA_MANAGES_ZK=true +fi + +if [ "$HAMA_MANAGES_ZK" = "true" ]; then + hosts=`"$bin"/hama org.apache.hama.zookeeper.ZKServerTool` + cmd=$"${@// /\\ }" + for zookeeper in $hosts; do + ssh $HAMA_SSH_OPTS $zookeeper $cmd 2>&1 | sed "s/^/$zookeeper: /" & + done +fi + +wait Propchange: incubator/hama/branches/HamaV2/core/bin/zookeepers.sh ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/conf/groomservers URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/conf/groomservers?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/conf/groomservers (added) +++ incubator/hama/branches/HamaV2/core/conf/groomservers Tue Sep 27 09:35:21 2011 @@ -0,0 +1 @@ +localhost \ No newline at end of file Added: incubator/hama/branches/HamaV2/core/conf/groomservers.template URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/conf/groomservers.template?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/conf/groomservers.template (added) +++ incubator/hama/branches/HamaV2/core/conf/groomservers.template Tue Sep 27 09:35:21 2011 @@ -0,0 +1 @@ +localhost \ No newline at end of file Added: incubator/hama/branches/HamaV2/core/conf/hama-default.xml URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/conf/hama-default.xml?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/conf/hama-default.xml (added) +++ incubator/hama/branches/HamaV2/core/conf/hama-default.xml Tue Sep 27 09:35:21 2011 @@ -0,0 +1,171 @@ + + + + + + bsp.master.address + local + The address of the bsp master server. Either the + literal string "local" or a host[:port] (where host is a name or + IP address) for distributed mode. + + + + bsp.master.port + 40000 + The port master should bind to. + + + bsp.peer.port + 61000 + The port an groom server binds to. + + + bsp.groom.rpc.port + 50000 + The port an groom rpc binds to. + + + bsp.http.infoserver.port + 40013 + The port where the web-interface can be seen. + + + + bsp.groom.report.address + 127.0.0.1:0 + The interface and port that groom server listens on. + Since it is only connected to by the tasks, it uses the local interface. + EXPERT ONLY. Should only be changed if your host does not have the loopback + interface. + + + bsp.system.dir + ${hadoop.tmp.dir}/bsp/system + The shared directory where BSP stores control files. + + + + bsp.local.dir + ${hadoop.tmp.dir}/bsp/local + local directory for temporal store. + + + hama.tmp.dir + /tmp/hama-${user.name} + Temporary directory on the local filesystem. + + + bsp.child.java.opts + -Xmx512m + Java opts for the groom server child processes. + The following symbol, if present, will be interpolated: @taskid@ is replaced + by current TaskID. Any other occurrences of '@' will go unchanged. + For example, to enable verbose gc logging to a file named for the taskid in + /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of: + -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc + The configuration variable bsp.child.ulimit can be used to control the + maximum virtual memory of the child processes. + + + + bsp.local.tasks.maximum + 20 + Number of tasks that run in parallel when in local mode. + + + bsp.tasks.maximum + 3 + The maximum number of BSP tasks that will be run simultaneously + by a groom server. + + + + + hama.zookeeper.property.initLimit + 10 + Property from ZooKeeper's config zoo.cfg. + The number of ticks that the initial synchronization phase can take. + + + + hama.zookeeper.property.syncLimit + 5 + Property from ZooKeeper's config zoo.cfg. + The number of ticks that can pass between sending a request and getting an + acknowledgment. + + + + hama.zookeeper.property.dataDir + ${hama.tmp.dir}/zookeeper + Property from ZooKeeper's config zoo.cfg. + The directory where the snapshot is stored. + + + + hama.zookeeper.property.clientPort + 21810 + Property from ZooKeeper's config zoo.cfg. + The port at which the clients will connect. + + + + hama.zookeeper.property.tickTime + 3000 + + + + hama.zookeeper.property.maxClientCnxns + 30 + Property from ZooKeeper's config zoo.cfg. + Limit on number of concurrent connections (at the socket level) that a + single client, identified by IP address, may make to a single member of + the ZooKeeper ensemble. Set high to avoid zk connection issues running + standalone and pseudo-distributed. + + + + + hama.zookeeper.quorum + localhost + Comma separated list of servers in the ZooKeeper quorum. + For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com". + By default this is set to localhost for local and pseudo-distributed modes + of operation. For a fully-distributed setup, this should be set to a full + list of ZooKeeper quorum servers. If HAMA_MANAGES_ZK is set in hama-env.sh + this is the list of servers which we will start/stop ZooKeeper on. + + + + + Propchange: incubator/hama/branches/HamaV2/core/conf/hama-default.xml ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/conf/hama-env.sh URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/conf/hama-env.sh?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/conf/hama-env.sh (added) +++ incubator/hama/branches/HamaV2/core/conf/hama-env.sh Tue Sep 27 09:35:21 2011 @@ -0,0 +1,34 @@ +# +#/** +# * Copyright 2007 The Apache Software Foundation +# * +# * Licensed to the Apache Software Foundation (ASF) under one +# * or more contributor license agreements. See the NOTICE file +# * distributed with this work for additional information +# * regarding copyright ownership. The ASF licenses this file +# * to you under the Apache License, Version 2.0 (the +# * "License"); you may not use this file except in compliance +# * with the License. You may obtain a copy of the License at +# * +# * http://www.apache.org/licenses/LICENSE-2.0 +# * +# * Unless required by applicable law or agreed to in writing, software +# * distributed under the License is distributed on an "AS IS" BASIS, +# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# * See the License for the specific language governing permissions and +# * limitations under the License. +# */ + +# Set environment variables here. + +# The java implementation to use. Required. +# export JAVA_HOME=/usr/lib/jvm/java-6-sun + +# Where log files are stored. $HAMA_HOME/logs by default. +# export HAMA_LOG_DIR=${HAMA_HOME}/logs + +# Extra ssh options. Empty by default. +# export HAMA_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HAMA_CONF_DIR" + +# Tell Hama whether it should manage it's own instance of Zookeeper or not. +# export HAMA_MANAGES_ZK=true Propchange: incubator/hama/branches/HamaV2/core/conf/hama-env.sh ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/conf/hama-env.sh.template URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/conf/hama-env.sh.template?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/conf/hama-env.sh.template (added) +++ incubator/hama/branches/HamaV2/core/conf/hama-env.sh.template Tue Sep 27 09:35:21 2011 @@ -0,0 +1,37 @@ +# +#/** +# * Copyright 2007 The Apache Software Foundation +# * +# * Licensed to the Apache Software Foundation (ASF) under one +# * or more contributor license agreements. See the NOTICE file +# * distributed with this work for additional information +# * regarding copyright ownership. The ASF licenses this file +# * to you under the Apache License, Version 2.0 (the +# * "License"); you may not use this file except in compliance +# * with the License. You may obtain a copy of the License at +# * +# * http://www.apache.org/licenses/LICENSE-2.0 +# * +# * Unless required by applicable law or agreed to in writing, software +# * distributed under the License is distributed on an "AS IS" BASIS, +# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# * See the License for the specific language governing permissions and +# * limitations under the License. +# */ + +# Set environment variables here. + +# The java implementation to use. Required. +# export JAVA_HOME=/usr/lib/jvm/java-6-sun + +# Where log files are stored. $HAMA_HOME/logs by default. +# export HAMA_LOG_DIR=${HAMA_HOME}/logs + +# The maximum amount of heap to use, in MB. Default is 1000. +# export HAMA_HEAPSIZE=1000 + +# Extra ssh options. Empty by default. +# export HAMA_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HAMA_CONF_DIR" + +# Tell Hama whether it should manage it's own instance of Zookeeper or not. +# export HAMA_MANAGES_ZK=true Added: incubator/hama/branches/HamaV2/core/conf/hama-site.xml URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/conf/hama-site.xml?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/conf/hama-site.xml (added) +++ incubator/hama/branches/HamaV2/core/conf/hama-site.xml Tue Sep 27 09:35:21 2011 @@ -0,0 +1,25 @@ + + + + + Propchange: incubator/hama/branches/HamaV2/core/conf/hama-site.xml ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/conf/hama-site.xml.template URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/conf/hama-site.xml.template?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/conf/hama-site.xml.template (added) +++ incubator/hama/branches/HamaV2/core/conf/hama-site.xml.template Tue Sep 27 09:35:21 2011 @@ -0,0 +1,25 @@ + + + + + Added: incubator/hama/branches/HamaV2/core/conf/log4j.properties URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/conf/log4j.properties?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/conf/log4j.properties (added) +++ incubator/hama/branches/HamaV2/core/conf/log4j.properties Tue Sep 27 09:35:21 2011 @@ -0,0 +1,81 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# Define some default values that can be overridden by system properties +hama.root.logger=INFO,console +hama.log.dir=. +hama.log.file=hama.log + +# Define the root logger to the system property "hama.root.logger". +log4j.rootLogger=${hama.root.logger} + +# Logging Threshold +log4j.threshhold=ALL + +# +# Daily Rolling File Appender +# +log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender +log4j.appender.DRFA.File=${hama.log.dir}/${hama.log.file} + +# Rollver at midnight +log4j.appender.DRFA.DatePattern=.yyyy-MM-dd + +# 30-day backup +#log4j.appender.DRFA.MaxBackupIndex=30 +log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout + +# Pattern format: Date LogLevel LoggerName LogMessage +log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n + +# Debugging Pattern format +#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n + +# +# TaskLog Appender +# + +#Default values +hama.tasklog.taskid=null +hama.tasklog.noKeepSplits=4 +hama.tasklog.totalLogFileSize=100 +hama.tasklog.purgeLogSplits=true +hama.tasklog.logsRetainHours=12 + +log4j.appender.TLA=org.apache.hama.bsp.TaskLogAppender +log4j.appender.TLA.taskId=${hama.tasklog.taskid} +log4j.appender.TLA.totalLogFileSize=${hama.tasklog.totalLogFileSize} + +log4j.appender.TLA.layout=org.apache.log4j.PatternLayout +log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n + +# +# console +# Add "console" to rootlogger above if you want to use this +# +log4j.appender.console=org.apache.log4j.ConsoleAppender +log4j.appender.console.target=System.err +log4j.appender.console.layout=org.apache.log4j.PatternLayout +log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n + +# Custom Logging levels + +#log4j.logger.org.apache.hadoop.fs.FSNamesystem=DEBUG +#log4j.logger.org.apache.hadoop.dfs=DEBUG +#log4j.logger.org.apache.hama=DEBUG Propchange: incubator/hama/branches/HamaV2/core/conf/log4j.properties ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/pom.xml URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/pom.xml?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/pom.xml (added) +++ incubator/hama/branches/HamaV2/core/pom.xml Tue Sep 27 09:35:21 2011 @@ -0,0 +1,137 @@ + + + + + + org.apache.hama + hama-parent + 0.4.0-incubating-SNAPSHOT + + + 4.0.0 + org.apache.hama + hama-core + Apache Hama Core + 0.4.0-incubating-SNAPSHOT + jar + + + scm:svn:http://svn.apache.org/repos/asf/incubator/hama/core + + + scm:svn:https://svn.apache.org/repos/asf/incubator/hama/core + + + + + + commons-logging + commons-logging + + + commons-cli + commons-cli + + + org.mortbay.jetty + jetty + + + org.apache.tomcat + servlet-api + + + org.mortbay.jetty + jetty-annotations + + + org.mortbay.jetty + jetty-util + + + org.mortbay.jetty + jsp-api-2.1 + + + org.mortbay.jetty + jsp-2.1 + + + org.apache.ant + ant + ${ant.version} + + + org.apache.ant + ant-launcher + ${ant.version} + + + org.apache.hadoop + hadoop-core + + + org.apache.hadoop + hadoop-test + + + junit + junit + test + + + org.slf4j + slf4j-log4j12 + + + log4j + log4j + provided + + + org.apache.zookeeper + zookeeper + + + + + + + org.apache.maven.plugins + maven-dependency-plugin + + + copy-dependencies + package + + copy-dependencies + + + ${basedir}/target/lib + false + true + true + + + + + + hama-core-${project.version} + + Propchange: incubator/hama/branches/HamaV2/core/pom.xml ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/BSPMasterRunner.java URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/BSPMasterRunner.java?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/BSPMasterRunner.java (added) +++ incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/BSPMasterRunner.java Tue Sep 27 09:35:21 2011 @@ -0,0 +1,60 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hama; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configured; +import org.apache.hadoop.util.StringUtils; +import org.apache.hadoop.util.Tool; +import org.apache.hadoop.util.ToolRunner; +import org.apache.hama.bsp.BSPMaster; + +/** + * This class starts and runs the BSPMaster. + */ +public class BSPMasterRunner extends Configured implements Tool { + + public static final Log LOG = LogFactory.getLog(BSPMasterRunner.class); + + @Override + public int run(String[] args) throws Exception { + StringUtils.startupShutdownMessage(BSPMaster.class, args, LOG); + + if (args.length != 0) { + System.out.println("usage: BSPMasterRunner"); + System.exit(-1); + } + + try { + HamaConfiguration conf = new HamaConfiguration(getConf()); + BSPMaster master = BSPMaster.startMaster(conf); + master.offerService(); + } catch (Throwable e) { + LOG.fatal(StringUtils.stringifyException(e)); + return -1; + } + return 0; + } + + public static void main(String[] args) throws Exception { + int exitCode = ToolRunner.run(new BSPMasterRunner(), args); + System.exit(exitCode); + } + +} Propchange: incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/BSPMasterRunner.java ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/Constants.java URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/Constants.java?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/Constants.java (added) +++ incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/Constants.java Tue Sep 27 09:35:21 2011 @@ -0,0 +1,97 @@ +/** + * Copyright 2007 The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hama; + +/** + * Some constants used in the Hama. + */ +public interface Constants { + + public static final String GROOM_RPC_HOST = "bsp.groom.rpc.hostname"; + + public static final String DEFAULT_GROOM_RPC_HOST = "0.0.0.0"; + + public static final String GROOM_RPC_PORT = "bsp.groom.rpc.port"; + + /** Default port region rpc server listens on. */ + public static final int DEFAULT_GROOM_RPC_PORT = 50000; + + + /////////////////////////////////////// + // Constants for BSP Package + /////////////////////////////////////// + /** default host address */ + public static final String PEER_HOST = "bsp.peer.hostname"; + /** default host address */ + public static final String DEFAULT_PEER_HOST = "0.0.0.0"; + + public static final String PEER_PORT = "bsp.peer.port"; + /** Default port region server listens on. */ + public static final int DEFAULT_PEER_PORT = 61000; + + public static final String PEER_ID = "bsp.peer.id"; + + /** Parameter name for what groom server implementation to use. */ + public static final String GROOM_SERVER_IMPL= "hama.groomserver.impl"; + + /** When we encode strings, we always specify UTF8 encoding */ + public static final String UTF8_ENCODING = "UTF-8"; + + public static final String MAX_TASKS_PER_GROOM = "bsp.tasks.maximum"; + + /////////////////////////////////////// + // Constants for ZooKeeper + /////////////////////////////////////// + /** zookeeper root */ + public static final String ZOOKEEPER_ROOT = "bsp.zookeeper.root"; + /** zookeeper default root */ + public static final String DEFAULT_ZOOKEEPER_ROOT = "/bsp"; + + /** zookeeper server address */ + public static final String ZOOKEEPER_SERVER_ADDRS = "zookeeper.server"; + /** zookeeper default server address */ + static final String DEFAULT_ZOOKEEPER_SERVER_ADDR = "localhost:21810"; + /** Parameter name for number of times to retry writes to ZooKeeper. */ + public static final String ZOOKEEPER_RETRIES = "zookeeper.retries"; + /** Default number of times to retry writes to ZooKeeper. */ + public static final int DEFAULT_ZOOKEEPER_RETRIES = 5; + /** Parameter name for ZooKeeper pause between retries. In milliseconds. */ + public static final String ZOOKEEPER_PAUSE = "zookeeper.pause"; + /** Default ZooKeeper pause value. In milliseconds. */ + public static final int DEFAULT_ZOOKEEPER_PAUSE = 2 * 1000; + + static final String ZOOKEEPER_CONFIG_NAME = "zoo.cfg"; + static final String ZOOKEEPER_CLIENT_PORT = "hama.zookeeper.property.clientPort"; + static final String ZOOKEEPER_SESSION_TIMEOUT = "hama.zookeeper.session.timeout"; + static final int DEFAULT_ZOOKEEPER_CLIENT_PORT = 21810; + static final String ZOOKEEPER_QUORUM = "hama.zookeeper.quorum"; + /** Cluster is in distributed mode or not */ + static final String CLUSTER_DISTRIBUTED = "hama.cluster.distributed"; + /** Cluster is fully-distributed */ + static final String CLUSTER_IS_DISTRIBUTED = "true"; + + + // Other constants + + /** + * An empty instance. + */ + static final byte [] EMPTY_BYTE_ARRAY = new byte [0]; +} Propchange: incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/Constants.java ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/GroomServerRunner.java URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/GroomServerRunner.java?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/GroomServerRunner.java (added) +++ incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/GroomServerRunner.java Tue Sep 27 09:35:21 2011 @@ -0,0 +1,63 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hama; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.conf.Configured; +import org.apache.hadoop.util.StringUtils; +import org.apache.hadoop.util.Tool; +import org.apache.hadoop.util.ToolRunner; +import org.apache.hama.bsp.GroomServer; + +/** + * This class starts and runs the GroomServer. + */ +public class GroomServerRunner extends Configured implements Tool { + + public static final Log LOG = LogFactory.getLog(GroomServerRunner.class); + + @Override + public int run(String[] args) throws Exception { + StringUtils.startupShutdownMessage(GroomServer.class, args, LOG); + + if (args.length != 0) { + System.out.println("usage: GroomServerRunner"); + System.exit(-1); + } + + try { + Configuration conf = new HamaConfiguration(getConf()); + GroomServer groom = GroomServer.constructGroomServer(GroomServer.class, + conf); + GroomServer.startGroomServer(groom).join(); + } catch (Throwable e) { + LOG.fatal(StringUtils.stringifyException(e)); + return -1; + } + + return 0; + } + + public static void main(String[] args) throws Exception { + int exitCode = ToolRunner.run(new GroomServerRunner(), args); + System.exit(exitCode); + } + +} Propchange: incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/GroomServerRunner.java ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/HamaConfiguration.java URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/HamaConfiguration.java?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/HamaConfiguration.java (added) +++ incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/HamaConfiguration.java Tue Sep 27 09:35:21 2011 @@ -0,0 +1,61 @@ +/** + * Copyright 2007 The Apache Software Foundation + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hama; + +import java.util.Map.Entry; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; + +/** + * Adds Hama configuration files to a Configuration. + */ +public class HamaConfiguration extends Configuration { + /** constructor */ + public HamaConfiguration() { + super(); + addHamaResources(); + } + + public HamaConfiguration(Path confFile) { + super(); + this.addResource(confFile); + } + + /** + * Create a clone of passed configuration. + * + * @param c Configuration to clone. + */ + public HamaConfiguration(final Configuration c) { + this(); + for (Entry e : c) { + set(e.getKey(), e.getValue()); + } + } + + /** + * Adds Hama configuration files to a Configuration + */ + private void addHamaResources() { + Configuration.addDefaultResource("hama-default.xml"); + Configuration.addDefaultResource("hama-site.xml"); + } +} Propchange: incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/HamaConfiguration.java ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/MiniZooKeeperCluster.java URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/MiniZooKeeperCluster.java?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/MiniZooKeeperCluster.java (added) +++ incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/MiniZooKeeperCluster.java Tue Sep 27 09:35:21 2011 @@ -0,0 +1,216 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hama; + +import java.io.BufferedReader; +import java.io.File; +import java.io.IOException; +import java.io.InputStreamReader; +import java.io.OutputStream; +import java.io.Reader; +import java.net.BindException; +import java.net.InetSocketAddress; +import java.net.Socket; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.FileUtil; +import org.apache.zookeeper.server.NIOServerCnxn; +import org.apache.zookeeper.server.ZooKeeperServer; +import org.apache.zookeeper.server.persistence.FileTxnLog; + +/** + * This class starts and runs the MiniZookeeperCluster. + */ +public class MiniZooKeeperCluster { + private static final Log LOG = LogFactory.getLog(MiniZooKeeperCluster.class); + + private static final int TICK_TIME = 2000; + private static final int CONNECTION_TIMEOUT = 30000; + + private boolean started; + private int clientPort = 21810; // use non-standard port + + private NIOServerCnxn.Factory standaloneServerFactory; + private int tickTime = 0; + + /** Create mini Zookeeper cluster. */ + public MiniZooKeeperCluster() { + this.started = false; + } + + public void setClientPort(int clientPort) { + this.clientPort = clientPort; + } + + public void setTickTime(int tickTime) { + this.tickTime = tickTime; + } + + private static void setupTestEnv() { + // during the tests we run with 100K prealloc in the logs. + // on windows systems prealloc of 64M was seen to take ~15seconds + // resulting in test failure (client timeout on first session). + // set env and directly in order to handle static init/gc issues + System.setProperty("zookeeper.preAllocSize", "100"); + FileTxnLog.setPreallocSize(100); + } + + /** + * @param baseDir + * @return ClientPort server bound to. + * @throws IOException + * @throws InterruptedException + */ + public int startup(File baseDir) throws IOException, + InterruptedException { + + setupTestEnv(); + + shutdown(); + + File dir = new File(baseDir, "zookeeper").getAbsoluteFile(); + recreateDir(dir); + + int tickTimeToUse; + if (this.tickTime > 0) { + tickTimeToUse = this.tickTime; + } else { + tickTimeToUse = TICK_TIME; + } + ZooKeeperServer server = new ZooKeeperServer(dir, dir, tickTimeToUse); + while (true) { + try { + standaloneServerFactory = + new NIOServerCnxn.Factory(new InetSocketAddress(clientPort)); + } catch (BindException e) { + LOG.info("Faild binding ZK Server to client port: " + clientPort); + //this port is already in use. try to use another + clientPort++; + continue; + } + break; + } + standaloneServerFactory.startup(server); + + if (!waitForServerUp(clientPort, CONNECTION_TIMEOUT)) { + throw new IOException("Waiting for startup of standalone server"); + } + + started = true; + + return clientPort; + } + + private void recreateDir(File dir) throws IOException { + if (dir.exists()) { + FileUtil.fullyDelete(dir); + } + try { + dir.mkdirs(); + } catch (SecurityException e) { + throw new IOException("creating dir: " + dir, e); + } + } + + /** + * @throws IOException + */ + public void shutdown() throws IOException { + if (!started) { + return; + } + + standaloneServerFactory.shutdown(); + if (!waitForServerDown(clientPort, CONNECTION_TIMEOUT)) { + throw new IOException("Waiting for shutdown of standalone server"); + } + + started = false; + } + + // XXX: From o.a.zk.t.ClientBase + private static boolean waitForServerDown(int port, long timeout) { + long start = System.currentTimeMillis(); + while (true) { + try { + Socket sock = new Socket("localhost", port); + try { + OutputStream outstream = sock.getOutputStream(); + outstream.write("stat".getBytes()); + outstream.flush(); + } finally { + sock.close(); + } + } catch (IOException e) { + return true; + } + + if (System.currentTimeMillis() > start + timeout) { + break; + } + try { + Thread.sleep(250); + } catch (InterruptedException e) { + // ignore + } + } + return false; + } + + // XXX: From o.a.zk.t.ClientBase + private static boolean waitForServerUp(int port, long timeout) { + long start = System.currentTimeMillis(); + while (true) { + try { + Socket sock = new Socket("localhost", port); + BufferedReader reader = null; + try { + OutputStream outstream = sock.getOutputStream(); + outstream.write("stat".getBytes()); + outstream.flush(); + + Reader isr = new InputStreamReader(sock.getInputStream()); + reader = new BufferedReader(isr); + String line = reader.readLine(); + if (line != null && line.startsWith("Zookeeper version:")) { + return true; + } + } finally { + sock.close(); + if (reader != null) { + reader.close(); + } + } + } catch (IOException e) { + // ignore as this is expected + LOG.info("server localhost:" + port + " not up " + e); + } + + if (System.currentTimeMillis() > start + timeout) { + break; + } + try { + Thread.sleep(250); + } catch (InterruptedException e) { + // ignore + } + } + return false; + } +} Propchange: incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/MiniZooKeeperCluster.java ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/ZooKeeperRunner.java URL: http://svn.apache.org/viewvc/incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/ZooKeeperRunner.java?rev=1176297&view=auto ============================================================================== --- incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/ZooKeeperRunner.java (added) +++ incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/ZooKeeperRunner.java Tue Sep 27 09:35:21 2011 @@ -0,0 +1,45 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hama; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configured; +import org.apache.hadoop.util.Tool; +import org.apache.hadoop.util.ToolRunner; +import org.apache.hama.zookeeper.QuorumPeer; + +/** + * This class starts and runs the ZooKeeperServer. + */ +public class ZooKeeperRunner extends Configured implements Tool { + + public static final Log LOG = LogFactory.getLog(ZooKeeperRunner.class); + + @Override + public int run(String[] args) throws Exception { + QuorumPeer.run(getConf()); + return 0; + } + + public static void main(String[] args) throws Exception { + int exitCode = ToolRunner.run(new ZooKeeperRunner(), args); + System.exit(exitCode); + } + +} Propchange: incubator/hama/branches/HamaV2/core/src/main/java/org/apache/hama/ZooKeeperRunner.java ------------------------------------------------------------------------------ svn:eol-style = native