hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vaibhav Puranik <vpura...@gmail.com>
Subject Re: Starting Hadoop cluster on EC2
Date Tue, 24 Nov 2009 22:58:37 GMT
Install a program called Jps.

It's a ps for java processes. execute jps, you should see the processes (if
they are running).

Also make sure you check your logs.

Regards,
Vaibhav

On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
mailinglists19@gmail.com> wrote:

> Awesome.  Thanks, Vaibhav.  Made progress.  No error messages from
> ./all-start.sh
>
> I will test to see if the cluster is deployed properly.
>
> Question:  On the slave, when I do:
>
> ps -eaf | grep 'hadoop'
>
> I don't see any processes.  Shouldn't I see 2 processes - one for Datanode
> and the other for TaskTracker?
>
> Please let me know.  Thanks again.
>
>
> On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <vpuranik@gmail.com>
> wrote:
>
> > Yes, I pretty much meant that.
> >
> > Add your master's public key to authorized_keys on slaves too.
> >
> > Regards,
> > Vaibhav
> >
> >
> >
> > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Sorry.. but not sure what you mean by... "In order to achieve that you
> > need
> > > to add the keypair  you are using to your
> > > auhtorized_keys file in the home dir/.ssh folder."
> > >
> > >
> > > I tried this....
> > >
> > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > >
> > > Is that what you meant by "add the keypair to authorized_keys"?
> > >
> > > Still can't ssh from Slave to Master.  It says.....  Permission denied
> > > (publickey).
> > >
> > >
> > > Sorry, I am a newbie when it comes to such environment issues.  Thanks
> > for
> > > your patience.  It's greatly appreciated.
> > >
> > >
> > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <vpuranik@gmail.com
> > > >wrote:
> > >
> > > > You need to be able to SSH without specifying the keypair from a
> slave
> > to
> > > > master (I think reverse it true too, but I am not sure).
> > > >
> > > > In order to achieve that you need to add the keypair  you are using
> to
> > > your
> > > > auhtorized_keys file in the home dir/.ssh folder.
> > > >
> > > > Once you setup passphraseless ssh, it should start working.
> Furthrmore,
> > > > after you set this up, try to ssh manully (without key) and if it
> asks
> > > you
> > > > the following message - say yes:
> > > >
> > > > The authenticity of host
> > > > 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)' can't
> > > > be established.
> > > > RSA key fingerprint is
> 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > Are you sure you want to continue connecting (yes/no)? yes
> > > > Warning: Permanently added
> > > > 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> > > > (RSA) to the list of known hosts.
> > > >
> > > > In other words, you have to make sure that the machine you are trying
> > to
> > > > ssh
> > > > is in your known_hosts file.
> > > >
> > > > Regards,
> > > > Vaibhav
> > > >
> > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > Thanks, Vaibhav.  That was helpful.  I am able to scp from my local
> > > > machine
> > > > > after I opened port 22 under my security group.
> > > > >
> > > > > But my cluster is still not starting as expected.  I mean... on my
> > > Master
> > > > > when I run:
> > > > >
> > > > > ./start-all.sh
> > > > >
> > > > > I get messages saying...
> > > > >
> > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > >
> > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > >
> > > > >
> > > > > I can ssh from my Slave to master if I do this:
> > > > >
> > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > >
> > > > >
> > > > > Any suggestions?  Thanks for your help.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> vpuranik@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > In order to use scp from your machine to an ec2 instance you
need
> > to
> > > > use
> > > > > > scp
> > > > > > -i and give it the path of your keypair.
> > > > > >
> > > > > > scp -i /path/my/ec2/keypair  <souce> <destination>
> > > > > >
> > > > > > Secondly in order to configure an ec2 cluster for Hadoop, you
> need
> > to
> > > > > make
> > > > > > a
> > > > > > security group (let's call it hadoop) and give hadoop access
to
> > > hadoop.
> > > > > >
> > > > > > You can read more about security groups here -
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > > > >
> > > > > > You also need to make sure that you can ssh from all the slaves
> to
> > > > master
> > > > > > without specifying password.
> > > > > >
> > > > > > Regards,
> > > > > > Vaibhav Puranik
> > > > > > GumGum
> > > > > >
> > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > > > mailinglists19@gmail.com> wrote:
> > > > > >
> > > > > > > Seems like I am not explaining my problem correctly.
> > > > > > >
> > > > > > > My Key Pair file is on my machine at work which is behind
a
> corp
> > > > > > firewall.
> > > > > > >  As such I can't 'scp' from the ec2 instance to my local
> machine
> > at
> > > > > work
> > > > > > to
> > > > > > > 'get' the file.  So I need a way to 'send' a file from
my
> machine
> > > to
> > > > > the
> > > > > > > ec2
> > > > > > > instance.  I tried using 'scp' (from my machine at work)
to the
> > ec2
> > > > > > machine
> > > > > > > but it says "Permission denied".  Does this make sense?
> > > > > > >
> > > > > > > May be I need to use the command line tools for EC2.  I
am
> > looking
> > > > into
> > > > > > > those right now, but if there's a better/easier way, please
let
> > me
> > > > > know.
> > > > > > >
> > > > > > > Thanks once again for your help.
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zjffdu@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > The ssh is installed on ec2 by default, otherwise
you have no
> > way
> > > > to
> > > > > > > login
> > > > > > > > to ec2
> > > > > > > >
> > > > > > > >
> > > > > > > > Jeff Zhang
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something
<
> > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy
> )
> > >  "It
> > > > > > does
> > > > > > > so
> > > > > > > > > by
> > > > > > > > > connecting to the host using SSH and there executes
an SCP
> > > server
> > > > > > > (scp)".
> > > > > > > > >
> > > > > > > > > So if SSH isn't working SCP won't work, either.
 In any
> case
> > I
> > > > > tried
> > > > > > to
> > > > > > > > scp
> > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > >
> > > > > > > > > Any other ideas?  Thanks.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > zjffdu@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > >
> > > > > > > > > > Jeff Zhang
> > > > > > > > > >
> > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something
Something <
> > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Trying to start a Hadoop cluster on
EC2.  (Yes,
> > Cloudera's
> > > > > > > > distribution
> > > > > > > > > > > works well, but trying to do this myself
so I can learn
> > > > what's
> > > > > > > > > happening
> > > > > > > > > > > behind the scene.)
> > > > > > > > > > >
> > > > > > > > > > > I have a Master & a Slave.  When
I start HDFS on the
> > > master,
> > > > I
> > > > > > get
> > > > > > > a
> > > > > > > > > > > message
> > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission
denied)" - where
> > 10.xxx
> > > is
> > > > > IP
> > > > > > > > > address
> > > > > > > > > > of
> > > > > > > > > > > the slave.
> > > > > > > > > > >
> > > > > > > > > > > Basic problem (I think) is that I can't
ssh from the
> > master
> > > > EC2
> > > > > > > > > instance
> > > > > > > > > > to
> > > > > > > > > > > the Slave EC2 instance.  What's the
best way to fix it?
> >  I
> > > > > think
> > > > > > I
> > > > > > > > need
> > > > > > > > > > the
> > > > > > > > > > > "Key Pair" file on my master.  I have
a key pair on my
> > > local
> > > > > > > machine,
> > > > > > > > > but
> > > > > > > > > > > how do I transfer it to the EC2 machine?
 (I know, I
> > know,
> > > I
> > > > > > > agree..
> > > > > > > > I
> > > > > > > > > am
> > > > > > > > > > > dumb :)  Should I FTP it?
> > > > > > > > > > >
> > > > > > > > > > > Please help.  Thanks.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message