hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Kostyrka <andr...@kostyrka.org>
Subject Re: Hadoop on EC2 for large cluster
Date Thu, 20 Mar 2008 18:10:44 GMT
Actually, I personally use the following "2 part" copy technique to copy
files to a cluster of boxes:

tar cf - myfile | dsh -f host-list-file -i -c -M tar xCfv /tmp -

The first tar packages myfile into a tar file.

dsh runs a tar that unpacks the tar (in the above case all boxes listed
in host-list-file would have a /tmp/myfile after the command).

Tar options that are relevant include C (chdir) and v (verbose, can be
given twice) so you see what got copied.

dsh options that are relevant:
-i copy stdin to all ssh processes, requires -c
-c do the ssh calls concurrently.
-M prefix the out from the ssh with the hostname.

While this is not rsync, it has the benefit of being processed
concurrently, and quite flexible.

Andreas

Am Donnerstag, den 20.03.2008, 19:57 +0200 schrieb Andrey Pankov:
> Hi,
> 
> Did you see hadoop-0.16.0/src/contrib/ec2/bin/start-hadoop script? It 
> already contains such part:
> 
> echo "Copying private key to slaves"
> for slave in `cat slaves`; do
>    scp $SSH_OPTS $PRIVATE_KEY_PATH "root@$slave:/root/.ssh/id_rsa"
>    ssh $SSH_OPTS "root@$slave" "chmod 600 /root/.ssh/id_rsa"
>    sleep 1
> done
> 
> Anyway, did you tried hadoop-ec2 script? It works well for task you 
> described.
> 
> 
> Prasan Ary wrote:
> > Hi All,
> >   I have been trying to configure Hadoop on EC2 for large number of clusters ( 100
plus). It seems that I have to copy EC2 private key to all the machines in the cluster so
that they can have SSH connections.
> >   For now it seems I have to run a script to copy the key file to each of the EC2
instances. I wanted to know if there is a better way to accomplish this.
> >    
> >   Thanks,
> >   PA
> > 
> >        
> > ---------------------------------
> > Never miss a thing.   Make Yahoo your homepage.
> 
> ---
> Andrey Pankov

Mime
View raw message