hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Kostyrka <andr...@kostyrka.org>
Subject Re: Hadoop on EC2 for large cluster
Date Thu, 20 Mar 2008 18:10:44 GMT
Actually, I personally use the following "2 part" copy technique to copy
files to a cluster of boxes:

tar cf - myfile | dsh -f host-list-file -i -c -M tar xCfv /tmp -

The first tar packages myfile into a tar file.

dsh runs a tar that unpacks the tar (in the above case all boxes listed
in host-list-file would have a /tmp/myfile after the command).

Tar options that are relevant include C (chdir) and v (verbose, can be
given twice) so you see what got copied.

dsh options that are relevant:
-i copy stdin to all ssh processes, requires -c
-c do the ssh calls concurrently.
-M prefix the out from the ssh with the hostname.

While this is not rsync, it has the benefit of being processed
concurrently, and quite flexible.


Am Donnerstag, den 20.03.2008, 19:57 +0200 schrieb Andrey Pankov:
> Hi,
> Did you see hadoop-0.16.0/src/contrib/ec2/bin/start-hadoop script? It 
> already contains such part:
> echo "Copying private key to slaves"
> for slave in `cat slaves`; do
>    scp $SSH_OPTS $PRIVATE_KEY_PATH "root@$slave:/root/.ssh/id_rsa"
>    ssh $SSH_OPTS "root@$slave" "chmod 600 /root/.ssh/id_rsa"
>    sleep 1
> done
> Anyway, did you tried hadoop-ec2 script? It works well for task you 
> described.
> Prasan Ary wrote:
> > Hi All,
> >   I have been trying to configure Hadoop on EC2 for large number of clusters ( 100
plus). It seems that I have to copy EC2 private key to all the machines in the cluster so
that they can have SSH connections.
> >   For now it seems I have to run a script to copy the key file to each of the EC2
instances. I wanted to know if there is a better way to accomplish this.
> >    
> >   Thanks,
> >   PA
> > 
> >        
> > ---------------------------------
> > Never miss a thing.   Make Yahoo your homepage.
> ---
> Andrey Pankov

View raw message