hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vaibhav Puranik <vpura...@gmail.com>
Subject Re: Starting Hadoop cluster on EC2
Date Tue, 24 Nov 2009 19:16:57 GMT
You need to be able to SSH without specifying the keypair from a slave to
master (I think reverse it true too, but I am not sure).

In order to achieve that you need to add the keypair  you are using to your
auhtorized_keys file in the home dir/.ssh folder.

Once you setup passphraseless ssh, it should start working. Furthrmore,
after you set this up, try to ssh manully (without key) and if it asks you
the following message - say yes:

The authenticity of host
'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)' can't
be established.
RSA key fingerprint is 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added
'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
(RSA) to the list of known hosts.

In other words, you have to make sure that the machine you are trying to ssh
is in your known_hosts file.

Regards,
Vaibhav

On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
mailinglists19@gmail.com> wrote:

> Thanks, Vaibhav.  That was helpful.  I am able to scp from my local machine
> after I opened port 22 under my security group.
>
> But my cluster is still not starting as expected.  I mean... on my Master
> when I run:
>
> ./start-all.sh
>
> I get messages saying...
>
> 10.252.xxx.xxx: Permission denied (publickey).
>
> where 10.252.xxx.xxx is the ip address of my Slave.
>
>
> I can ssh from my Slave to master if I do this:
>
> ssh -i <keypairname>.pem <Master's ip address>
>
>
> Any suggestions?  Thanks for your help.
>
>
>
>
> On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <vpuranik@gmail.com>
> wrote:
>
> > In order to use scp from your machine to an ec2 instance you need to use
> > scp
> > -i and give it the path of your keypair.
> >
> > scp -i /path/my/ec2/keypair  <souce> <destination>
> >
> > Secondly in order to configure an ec2 cluster for Hadoop, you need to
> make
> > a
> > security group (let's call it hadoop) and give hadoop access to hadoop.
> >
> > You can read more about security groups here -
> >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> >
> > You also need to make sure that you can ssh from all the slaves to master
> > without specifying password.
> >
> > Regards,
> > Vaibhav Puranik
> > GumGum
> >
> > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Seems like I am not explaining my problem correctly.
> > >
> > > My Key Pair file is on my machine at work which is behind a corp
> > firewall.
> > >  As such I can't 'scp' from the ec2 instance to my local machine at
> work
> > to
> > > 'get' the file.  So I need a way to 'send' a file from my machine to
> the
> > > ec2
> > > instance.  I tried using 'scp' (from my machine at work) to the ec2
> > machine
> > > but it says "Permission denied".  Does this make sense?
> > >
> > > May be I need to use the command line tools for EC2.  I am looking into
> > > those right now, but if there's a better/easier way, please let me
> know.
> > >
> > > Thanks once again for your help.
> > >
> > >
> > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zjffdu@gmail.com> wrote:
> > >
> > > > The ssh is installed on ec2 by default, otherwise you have no way to
> > > login
> > > > to ec2
> > > >
> > > >
> > > > Jeff Zhang
> > > >
> > > >
> > > >
> > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy)  "It
> > does
> > > so
> > > > > by
> > > > > connecting to the host using SSH and there executes an SCP server
> > > (scp)".
> > > > >
> > > > > So if SSH isn't working SCP won't work, either.  In any case I
> tried
> > to
> > > > scp
> > > > > but getting "Permission denied (public key)".
> > > > >
> > > > > Any other ideas?  Thanks.
> > > > >
> > > > >
> > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zjffdu@gmail.com>
> > wrote:
> > > > >
> > > > > > You should scp the key-pair to EC2 machine
> > > > > >
> > > > > > Jeff Zhang
> > > > > >
> > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > > mailinglists19@gmail.com> wrote:
> > > > > >
> > > > > > > Trying to start a Hadoop cluster on EC2.  (Yes, Cloudera's
> > > > distribution
> > > > > > > works well, but trying to do this myself so I can learn
what's
> > > > > happening
> > > > > > > behind the scene.)
> > > > > > >
> > > > > > > I have a Master & a Slave.  When I start HDFS on the
master, I
> > get
> > > a
> > > > > > > message
> > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx
is
> IP
> > > > > address
> > > > > > of
> > > > > > > the slave.
> > > > > > >
> > > > > > > Basic problem (I think) is that I can't ssh from the master
EC2
> > > > > instance
> > > > > > to
> > > > > > > the Slave EC2 instance.  What's the best way to fix it?
 I
> think
> > I
> > > > need
> > > > > > the
> > > > > > > "Key Pair" file on my master.  I have a key pair on my
local
> > > machine,
> > > > > but
> > > > > > > how do I transfer it to the EC2 machine?  (I know, I know,
I
> > > agree..
> > > > I
> > > > > am
> > > > > > > dumb :)  Should I FTP it?
> > > > > > >
> > > > > > > Please help.  Thanks.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message