hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Something Something <mailinglist...@gmail.com>
Subject Re: Starting Hadoop cluster on EC2
Date Tue, 24 Nov 2009 18:54:30 GMT
Thanks, Vaibhav.  That was helpful.  I am able to scp from my local machine
after I opened port 22 under my security group.

But my cluster is still not starting as expected.  I mean... on my Master
when I run:

./start-all.sh

I get messages saying...

10.252.xxx.xxx: Permission denied (publickey).

where 10.252.xxx.xxx is the ip address of my Slave.


I can ssh from my Slave to master if I do this:

ssh -i <keypairname>.pem <Master's ip address>


Any suggestions?  Thanks for your help.




On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <vpuranik@gmail.com> wrote:

> In order to use scp from your machine to an ec2 instance you need to use
> scp
> -i and give it the path of your keypair.
>
> scp -i /path/my/ec2/keypair  <souce> <destination>
>
> Secondly in order to configure an ec2 cluster for Hadoop, you need to make
> a
> security group (let's call it hadoop) and give hadoop access to hadoop.
>
> You can read more about security groups here -
>
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
>
> You also need to make sure that you can ssh from all the slaves to master
> without specifying password.
>
> Regards,
> Vaibhav Puranik
> GumGum
>
> On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Seems like I am not explaining my problem correctly.
> >
> > My Key Pair file is on my machine at work which is behind a corp
> firewall.
> >  As such I can't 'scp' from the ec2 instance to my local machine at work
> to
> > 'get' the file.  So I need a way to 'send' a file from my machine to the
> > ec2
> > instance.  I tried using 'scp' (from my machine at work) to the ec2
> machine
> > but it says "Permission denied".  Does this make sense?
> >
> > May be I need to use the command line tools for EC2.  I am looking into
> > those right now, but if there's a better/easier way, please let me know.
> >
> > Thanks once again for your help.
> >
> >
> > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zjffdu@gmail.com> wrote:
> >
> > > The ssh is installed on ec2 by default, otherwise you have no way to
> > login
> > > to ec2
> > >
> > >
> > > Jeff Zhang
> > >
> > >
> > >
> > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy)  "It
> does
> > so
> > > > by
> > > > connecting to the host using SSH and there executes an SCP server
> > (scp)".
> > > >
> > > > So if SSH isn't working SCP won't work, either.  In any case I tried
> to
> > > scp
> > > > but getting "Permission denied (public key)".
> > > >
> > > > Any other ideas?  Thanks.
> > > >
> > > >
> > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zjffdu@gmail.com>
> wrote:
> > > >
> > > > > You should scp the key-pair to EC2 machine
> > > > >
> > > > > Jeff Zhang
> > > > >
> > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > mailinglists19@gmail.com> wrote:
> > > > >
> > > > > > Trying to start a Hadoop cluster on EC2.  (Yes, Cloudera's
> > > distribution
> > > > > > works well, but trying to do this myself so I can learn what's
> > > > happening
> > > > > > behind the scene.)
> > > > > >
> > > > > > I have a Master & a Slave.  When I start HDFS on the master,
I
> get
> > a
> > > > > > message
> > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is
IP
> > > > address
> > > > > of
> > > > > > the slave.
> > > > > >
> > > > > > Basic problem (I think) is that I can't ssh from the master
EC2
> > > > instance
> > > > > to
> > > > > > the Slave EC2 instance.  What's the best way to fix it?  I think
> I
> > > need
> > > > > the
> > > > > > "Key Pair" file on my master.  I have a key pair on my local
> > machine,
> > > > but
> > > > > > how do I transfer it to the EC2 machine?  (I know, I know, I
> > agree..
> > > I
> > > > am
> > > > > > dumb :)  Should I FTP it?
> > > > > >
> > > > > > Please help.  Thanks.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message