hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Lucene-hadoop Wiki] Update of "AmazonEC2" by TomWhite
Date Thu, 18 Jan 2007 09:42:41 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.

The following page has been changed by TomWhite:

  [http://www.amazon.com/gp/browse.html?node=201590011 Amazon EC2] (Elastic Compute Cloud)
is a computing service.  One allocates a set of hosts, and runs ones's application on them,
then, when done, de-allocates the hosts.  Billing is hourly per host.  Thus EC2 permits one
to deploy Hadoop on a cluster without having to own and operate that cluster, but rather renting
it on an hourly basis.
  This document assumes that you have already followed the steps in [http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-06-26/
Amazon's Getting Started Guide].
+ There are now some [#AutomatedScripts scripts] available for running Hadoop on EC2.
  == Concepts ==
@@ -222, +224 @@

  Please vote for these issues in Jira if you feel this would help your project.  (Anyone
can create themselves a Jira account in order to vote on issues, etc.)
+ [[Anchor(AutomatedScripts])]]
+ = Automated Scripts =
+ == Setting up ==
+  * Make sure you've followed the Amazon EC2 [http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-10-01/
Getting Started Guide], sections "Setting up an Account", "Setting up the Tools" and the "Generating
a Keypair" section of "Running an Instance".
+  * Unpack the Hadoop EC2 scripts distribution available from [http://issues.apache.org/jira/browse/HADOOP-884
+  * Edit all relevant variables in `hadoop-ec2-env.sh`.
+    * You need to get a Java download URL by visiting [http://java.sun.com/javase/downloads/index_jdk5.jsp
here]. Make sure you get the JDK (not JRE) labelled "Linux self-extracting file". (The scripts
have not been tested with Java 6 yet.)
+ == Creating an image ==
+ You only need to do this once.
+  * Type {{{
+ ./create-hadoop-image
+ }}}
+  * Accept the Java license terms.
+  * The script will create a new image, then bundle, upload and register it. This may take
some time. Be patient - don't assume it's crashed.
+  * Terminate you instance using the command given by the script.
+ == Running a job on a cluster ==
+  * Run cluster. During execution of this script you will be prompted to set up DNS (e.g.
[http://www.dyndns.com/services/dns/dyndns/ DynDNS]). {{{
+ ./run-hadoop-cluster 
+ }}}
+  * You will then be logged into the master node where you can start your job.
+    * For example, to test your cluster, try {{{
+ cd /usr/local/hadoop-*
+ bin/hadoop jar hadoop-*-examples.jar pi 10 10000000
+ }}}
+  * You can check progress of your job at `http://MASTER_HOST:50030/`.
+  * When you have finished logout of the master node by typing `exit`, then shutdown the
cluster with {{{
+ ./terminate-hadoop-cluster
+ }}}

View raw message