hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-2410) Make EC2 cluster nodes more independent of each other
Date Fri, 04 Apr 2008 13:53:24 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tom White updated HADOOP-2410:
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.17.0
         Assignee: Chris K Wensel
     Release Note: The command "hadoop-ec2 run" has been replaced by "hadoop-ec2 launch-cluster
<group> <number of instances>", and "hadoop-ec2 start-hadoop" has been removed
since Hadoop is started on instance start up. See http://wiki.apache.org/hadoop/AmazonEC2
for details.
     Hadoop Flags: [Incompatible change, Reviewed]
           Status: Resolved  (was: Patch Available)

I've just committed this. Thanks Chris!

I tried out the new scripts and they worked fine. I changed the version of Hadoop in the env
file to be 0.17.0 so that it picks up the new AMI when it is created (after 0.17.0 is released).
I also changed the version of Java to 1.6.0_05.

Chris, could you update the documentation on the wiki page with the changes please? It would
be worth keeping the instructions for the older scripts around on the same page.

> Make EC2 cluster nodes more independent of each other
> -----------------------------------------------------
>
>                 Key: HADOOP-2410
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2410
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/ec2
>    Affects Versions: 0.16.1
>            Reporter: Tom White
>            Assignee: Chris K Wensel
>             Fix For: 0.17.0
>
>         Attachments: concurrent-clusters-2.patch, concurrent-clusters-3.patch, concurrent-clusters.patch,
ec2.tgz
>
>
> The cluster start up scripts currently wait for each node to start up before appointing
a master (to run the namenode and jobtracker on), and copying private keys to all the nodes,
and writing the private IP address of the master to the hadoop-site.xml file (which is then
copied to the slaves via rsync). Only once this is all done is hadoop started on the cluster
(from the master). This can fail if any of the nodes fails to come up, which can happen as
EC2 doesn't guarantee that you get a cluster of the size you ask for (I've seen this happen).
> The process would be more robust if each node was told the address of the master as user
metadata and then started its own daemons. This is complicated by the fact that the public
DNS alias of the master resolves to a public IP address so cannot be used by EC2 nodes (see
http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/instance-addressing.html).
Instead we need to use a trick (http://developer.amazonwebservices.com/connect/message.jspa?messageID=71126#71126)
to find the private IP, and what's more we need to attempt to resolve the private IP in a
loop until it is available since the DNS will only be set up after the master has started.
> This change will also mean the private key doesn't need to be copied to each node, which
can be slow and has dubious security. Configuration can be handled using the mechanism described
in HADOOP-2409.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message