hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James P. White (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-952) Create a public (shared) Hadoop EC2 AMI
Date Sat, 10 Feb 2007 08:40:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12471973
] 

James P. White commented on HADOOP-952:
---------------------------------------

Hi Tom!

You wrote:

> ...
> Any problems or questions, give me a shout! (Let me know how it goes 
> anyway.)

I've gotten setup on EC2 and gave your image a whirl.

The biggest problem I had was figuring out the S3_BUCKET.  

I got HADOOP_VERSION wrong a couple times.

I also spent a while getting the EC2_KEYDIR and SSH_OPTS set to use my scheme.

These are the settings I wound up with:

# The Amazon S3 bucket where the Hadoop AMI you create will be stored.
S3_BUCKET=hadoop-ec2-images

# Location of EC2 keys.
# The default setting is probably OK if you set up EC2 following the Amazon Getting Started
guide.
EC2_KEYDIR=`dirname "$EC2_PRIVATE_KEY"`

# SSH options used when connecting to EC2 instances.
# Change the -i option to be the absolute path to your keypair that you set up in the Amazon
Getting Started guide.
SSH_OPTS=`echo -i "$EC2_KEYDIR"/id_rsa-gsg-keypair -o StrictHostKeyChecking=no`

# The download URL for the Sun JDK. Visit http://java.sun.com/javase/downloads/index_jdk5.jsp
and get the URL for the "Linux self-extracting file".
JAVA_BINARY_URL=''

# The version number of the installed JDK.
JAVA_VERSION=1.5.0_11

# The EC2 group to run your cluster in.
GROUP=hadoop-cluster-group

# The version of Hadoop to install.
HADOOP_VERSION=0.11.0

I think those are somewhat better defaults.  The others are much more self-explanatory.

I also had to rerun the run-cluster code following the "Waiting before ..." point multiple
times to get the settings worked out, so I made a shortened version (rerun-).  I also made
a login script (which turns out to be a good test before doing the "Creating instances...
business").

I then tried to run the pi sample job per the wiki page, but get an exception:

[root@domU-12-31-34-00-03-2F ~]# cd /usr/local/hadoop-0.11.0/
[root@domU-12-31-34-00-03-2F hadoop-0.11.0]# bin/hadoop jar hadoop-0.11.0-examples.jar pi
10 10000000
Number of Maps = 10 Samples per Map = 10000000
org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.ArithmeticException:
/ by zero
        at org.apache.hadoop.dfs.FSNamesystem$Replicator.chooseTarget(FSNamesystem.java:2593)
        at org.apache.hadoop.dfs.FSNamesystem$Replicator.chooseTarget(FSNamesystem.java:2555)
        at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:684)
        at org.apache.hadoop.dfs.NameNode.create(NameNode.java:248)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)

        at org.apache.hadoop.ipc.Client.call(Client.java:467)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:164)
        at org.apache.hadoop.dfs.$Proxy0.create(Unknown Source)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateNewBlock(DFSClient.java:1091)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:1031)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.endBlock(DFSClient.java:1255)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1345)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
        at org.apache.hadoop.fs.FSDataOutputStream$Summer.close(FSDataOutputStream.java:98)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
        at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:724)
        at org.apache.hadoop.examples.PiEstimator.launch(PiEstimator.java:185)
        at org.apache.hadoop.examples.PiEstimator.main(PiEstimator.java:226)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:143)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:40)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
[root@domU-12-31-34-00-03-2F hadoop-0.11.0]# 



> Create a public (shared) Hadoop EC2 AMI
> ---------------------------------------
>
>                 Key: HADOOP-952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-952
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.11.0
>            Reporter: Tom White
>         Assigned To: Tom White
>         Attachments: hadoop-952.patch
>
>
> HADOOP-884 makes it easy to run Hadoop on an EC2 cluster, but building an AMI (Abstract
Machine Image) can take a little while. Amazon EC2 supports shared AMIs (http://developer.amazonwebservices.com/connect/entry.jspa?entryID=530&ref=featured),
so we could provide publically available AMIs for each Hadoop release.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message