hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Something Something <mailinglist...@gmail.com>
Subject Re: Starting Hadoop cluster on EC2
Date Mon, 30 Nov 2009 07:52:05 GMT
Everything started working after I added the following line to hadoop-env.sh

export HADOOP_SSH_OPTS="$HADOOP_SSH_OPTS -i /mnt/myVol/KeyPairName.pem "

Thanks for your help.


On Wed, Nov 25, 2009 at 10:25 AM, Vaibhav Puranik <vpuranik@gmail.com>wrote:

> On master, in .ssh folder, there should be a file id_dsa.pub. This is
> master's public key.
>
> Contents of this file should be copied to slave's authorized_keys file.
>
> Regards,
> Vaibhav
>
> On Wed, Nov 25, 2009 at 9:45 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Can you please explain what you mean by "add master's public key to
> slave's
> > authorized_keys"?  I have a feeling I am not doing this correctly.  What
> I
> > did is this:
> >
> > 1)  On Master, opened ~/.ssh/authorized_key and copied lines that look
> like
> > this...
> >
> > ssh-dss
> >
> >
> AAAAB3NzaC1kc3MAAACBAKJOife8p+6vjJ/jODSWaBgyawN7XWmasasasb0cVqhe1WOpnJQpXuJeqsaBaZYHYxj8P74m1rOIdN9lSPAz6gEjHo4ft8UM7ZVExRC6HDM7JdfdferhwRgQSVI9ruTLlssO9i1D17mBnLotsgy92CyS7bsanM25nRqhmvqkX7tuKp66nlAAAAFQCnbwPn3v+ttorRZ4OijkAKoJaVFQAAAIAGVyZGwkm5DwYvige0VYhXZns8qSvW15dwSrS1a6k41q3C91AQZylvJYDnBjNGnWwUHfyWpyruzVCOskavEmsl6FcouU5Av1zLRT1sk4llUzZxXFtv6gzRBq2+aiCcoex3O+RRJbHa89cMLy6jeZOCFahwVz6IyHcQ0b7gZjQgFQAAAIBA4vzSDy5oN27Ji3mF9ZklsSmlJJrrJxGJfOn6/BK5tyvBijLy6tEJHrH3dY8TEJUp2V9RINKwcsy5LdqyI5gJRWoM8WdNzmUoIOLAW0HIVl2MldKKqP/ctQwGI94EjykSUYVt/IAvUcjFuIrtiGrMDjsqfm7h1Gi2ZtDQ==
> > root@domU-12-31-38-00-39-C8
> >
> >
> > to the end of ~/.ssh/authorized_key on Slave.
> >
> > Something tells me there's a better way to do this.  Is there?
> >
> >
> > On Wed, Nov 25, 2009 at 9:21 AM, Vaibhav Puranik <vpuranik@gmail.com>
> > wrote:
> >
> > > Actually I was wrong about adding keypair. It should be resolved by
> > adding
> > > master's public key to slave's authorized_keys. Amazon puts your
> keypair
> > in
> > > authroized_keys automatically. I missed that part.
> > >
> > > Regards,
> > > Vaibhav
> > >
> > > On Wed, Nov 25, 2009 at 8:38 AM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > Me again :(
> > > >
> > > > I decided to start fresh so that I can document the steps, but now I
> > > can't
> > > > even go further than I did before :(
> > > >
> > > > Please let me know what step I am missing:
> > > >
> > > > 1)  Launched 2 instances on AWS console - one for Master, one for
> > Slave.
> > > >
> > > > 2)  In one command window, connected to instance 1:
> > > > ssh -i MyKeyPair.pem <instance 1>
> > > > (Got 'authenticity' warning.. typed 'yes')
> > > >
> > > > This is my "Master" window.
> > > >
> > > > 3)  Opened another command window & connected to instance 2:
> > > > ssh -i MyKeyPair.pem <instance 2>
> > > > (Got 'authenticity' warning.. typed 'yes')
> > > >
> > > > This is my "Slave" window.
> > > >
> > > > 4) Opened another command window and 'scp' the key pair (from my
> local
> > > > machine) to instance 1 & instance 2.
> > > > scp -i MyKeyPair.pem MyKeyPair.pem <master instance>:/tmp
> > > > scp -i MyKeyPair.pem MyKeyPair.pem <slave instance>:/tmp
> > > >
> > > > 5)  On BOTH (Master & Slave) added Key Pair to 'authorized_key'
> > > >
> > > > cat /tmp/MyKeyPair.pem >> ~/.ssh/authorized_keys
> > > >
> > > > 6)  To setup passphraseless ssh ran these on BOTH - Master & Slave:
> > > > ssh localhost  (Got 'authenticiy' message.  Typed 'yes'.  Got
> > Permission
> > > > denied (public key)
> > > > ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
> > > > cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
> > > >
> > > > 7)  In order to ssh from Master to Slave (without entering KeyPair),
> > > copied
> > > > the key (that starts with 'sh-dss') from Slave's 'authorized_key'
> file
> > to
> > > > Master's 'authoried_key' file.
> > > >
> > > > Now, every time I try (from Master):
> > > >
> > > > ssh <slave's ip address>
> > > >
> > > > I get - "Permission denied (publickey).
> > > >
> > > > But, when I type:
> > > >
> > > > ssh -i /tmp/MyKeyPair.pem <slave's ip address>
> > > >
> > > > It works.
> > > >
> > > > I thought step #5 above should have resolved this.  What am I doing
> > > wrong?
> > > > I believe, in order for me to start the cluster, I should be able to
> > > 'ssh'
> > > > to slave without using key pair, correct?
> > > >
> > > >
> > > > Thanks for your help.
> > > >
> > > >
> > > > On Tue, Nov 24, 2009 at 2:58 PM, Vaibhav Puranik <vpuranik@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Install a program called Jps.
> > > > >
> > > > > It's a ps for java processes. execute jps, you should see the
> > processes
> > > > (if
> > > > > they are running).
> > > > >
> > > > > Also make sure you check your logs.
> > > > >
> > > > > Regards,
> > > > > Vaibhav
> > > > >
> > > > > On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
> > > > > mailinglists19@gmail.com> wrote:
> > > > >
> > > > > > Awesome.  Thanks, Vaibhav.  Made progress.  No error messages
> from
> > > > > > ./all-start.sh
> > > > > >
> > > > > > I will test to see if the cluster is deployed properly.
> > > > > >
> > > > > > Question:  On the slave, when I do:
> > > > > >
> > > > > > ps -eaf | grep 'hadoop'
> > > > > >
> > > > > > I don't see any processes.  Shouldn't I see 2 processes - one
for
> > > > > Datanode
> > > > > > and the other for TaskTracker?
> > > > > >
> > > > > > Please let me know.  Thanks again.
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <
> > vpuranik@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Yes, I pretty much meant that.
> > > > > > >
> > > > > > > Add your master's public key to authorized_keys on slaves
too.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Vaibhav
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > >
> > > > > > > > Sorry.. but not sure what you mean by... "In order
to achieve
> > > that
> > > > > you
> > > > > > > need
> > > > > > > > to add the keypair  you are using to your
> > > > > > > > auhtorized_keys file in the home dir/.ssh folder."
> > > > > > > >
> > > > > > > >
> > > > > > > > I tried this....
> > > > > > > >
> > > > > > > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > > > > > > >
> > > > > > > > Is that what you meant by "add the keypair to
> authorized_keys"?
> > > > > > > >
> > > > > > > > Still can't ssh from Slave to Master.  It says.....
>  Permission
> > > > > denied
> > > > > > > > (publickey).
> > > > > > > >
> > > > > > > >
> > > > > > > > Sorry, I am a newbie when it comes to such environment
> issues.
> > > > >  Thanks
> > > > > > > for
> > > > > > > > your patience.  It's greatly appreciated.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik
<
> > > > > vpuranik@gmail.com
> > > > > > > > >wrote:
> > > > > > > >
> > > > > > > > > You need to be able to SSH without specifying
the keypair
> > from
> > > a
> > > > > > slave
> > > > > > > to
> > > > > > > > > master (I think reverse it true too, but I am
not sure).
> > > > > > > > >
> > > > > > > > > In order to achieve that you need to add the
keypair  you
> are
> > > > using
> > > > > > to
> > > > > > > > your
> > > > > > > > > auhtorized_keys file in the home dir/.ssh folder.
> > > > > > > > >
> > > > > > > > > Once you setup passphraseless ssh, it should
start working.
> > > > > > Furthrmore,
> > > > > > > > > after you set this up, try to ssh manully (without
key) and
> > if
> > > it
> > > > > > asks
> > > > > > > > you
> > > > > > > > > the following message - say yes:
> > > > > > > > >
> > > > > > > > > The authenticity of host
> > > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com
> > (147.127.186.243)'
> > > > > can't
> > > > > > > > > be established.
> > > > > > > > > RSA key fingerprint is
> > > > > > 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > > > > > > Are you sure you want to continue connecting
(yes/no)? yes
> > > > > > > > > Warning: Permanently added
> > > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com
> > ,147.127.186.243'
> > > > > > > > > (RSA) to the list of known hosts.
> > > > > > > > >
> > > > > > > > > In other words, you have to make sure that the
machine you
> > are
> > > > > trying
> > > > > > > to
> > > > > > > > > ssh
> > > > > > > > > is in your known_hosts file.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Vaibhav
> > > > > > > > >
> > > > > > > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something
<
> > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Thanks, Vaibhav.  That was helpful.  I am
able to scp
> from
> > my
> > > > > local
> > > > > > > > > machine
> > > > > > > > > > after I opened port 22 under my security
group.
> > > > > > > > > >
> > > > > > > > > > But my cluster is still not starting as
expected.  I
> > mean...
> > > on
> > > > > my
> > > > > > > > Master
> > > > > > > > > > when I run:
> > > > > > > > > >
> > > > > > > > > > ./start-all.sh
> > > > > > > > > >
> > > > > > > > > > I get messages saying...
> > > > > > > > > >
> > > > > > > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > > > > > > >
> > > > > > > > > > where 10.252.xxx.xxx is the ip address of
my Slave.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I can ssh from my Slave to master if I do
this:
> > > > > > > > > >
> > > > > > > > > > ssh -i <keypairname>.pem <Master's
ip address>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Any suggestions?  Thanks for your help.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav
Puranik <
> > > > > > vpuranik@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > In order to use scp from your machine
to an ec2
> instance
> > > you
> > > > > need
> > > > > > > to
> > > > > > > > > use
> > > > > > > > > > > scp
> > > > > > > > > > > -i and give it the path of your keypair.
> > > > > > > > > > >
> > > > > > > > > > > scp -i /path/my/ec2/keypair  <souce>
<destination>
> > > > > > > > > > >
> > > > > > > > > > > Secondly in order to configure an ec2
cluster for
> Hadoop,
> > > you
> > > > > > need
> > > > > > > to
> > > > > > > > > > make
> > > > > > > > > > > a
> > > > > > > > > > > security group (let's call it hadoop)
and give hadoop
> > > access
> > > > to
> > > > > > > > hadoop.
> > > > > > > > > > >
> > > > > > > > > > > You can read more about security groups
here -
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > > > > > > > > >
> > > > > > > > > > > You also need to make sure that you
can ssh from all
> the
> > > > slaves
> > > > > > to
> > > > > > > > > master
> > > > > > > > > > > without specifying password.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Vaibhav Puranik
> > > > > > > > > > > GumGum
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something
Something <
> > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Seems like I am not explaining
my problem correctly.
> > > > > > > > > > > >
> > > > > > > > > > > > My Key Pair file is on my machine
at work which is
> > behind
> > > a
> > > > > > corp
> > > > > > > > > > > firewall.
> > > > > > > > > > > >  As such I can't 'scp' from the
ec2 instance to my
> > local
> > > > > > machine
> > > > > > > at
> > > > > > > > > > work
> > > > > > > > > > > to
> > > > > > > > > > > > 'get' the file.  So I need a way
to 'send' a file
> from
> > my
> > > > > > machine
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > ec2
> > > > > > > > > > > > instance.  I tried using 'scp'
(from my machine at
> > work)
> > > to
> > > > > the
> > > > > > > ec2
> > > > > > > > > > > machine
> > > > > > > > > > > > but it says "Permission denied".
 Does this make
> sense?
> > > > > > > > > > > >
> > > > > > > > > > > > May be I need to use the command
line tools for EC2.
>  I
> > > am
> > > > > > > looking
> > > > > > > > > into
> > > > > > > > > > > > those right now, but if there's
a better/easier way,
> > > please
> > > > > let
> > > > > > > me
> > > > > > > > > > know.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks once again for your help.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:19 AM,
Jeff Zhang <
> > > > > zjffdu@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > The ssh is installed on ec2
by default, otherwise
> you
> > > > have
> > > > > no
> > > > > > > way
> > > > > > > > > to
> > > > > > > > > > > > login
> > > > > > > > > > > > > to ec2
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:09
AM, Something
> Something
> > <
> > > > > > > > > > > > > mailinglists19@gmail.com>
wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > As per Wikipedia (
> > > > > http://en.wikipedia.org/wiki/Secure_copy
> > > > > > )
> > > > > > > >  "It
> > > > > > > > > > > does
> > > > > > > > > > > > so
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > connecting to the host
using SSH and there
> executes
> > > an
> > > > > SCP
> > > > > > > > server
> > > > > > > > > > > > (scp)".
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > So if SSH isn't working
SCP won't work, either.
>  In
> > > any
> > > > > > case
> > > > > > > I
> > > > > > > > > > tried
> > > > > > > > > > > to
> > > > > > > > > > > > > scp
> > > > > > > > > > > > > > but getting "Permission
denied (public key)".
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Any other ideas?  Thanks.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sat, Nov 21, 2009
at 6:12 PM, Jeff Zhang <
> > > > > > > zjffdu@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > You should scp
the key-pair to EC2 machine
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sat, Nov 21,
2009 at 4:57 PM, Something
> > > Something
> > > > <
> > > > > > > > > > > > > > > mailinglists19@gmail.com>
wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Trying to
start a Hadoop cluster on EC2.
>  (Yes,
> > > > > > > Cloudera's
> > > > > > > > > > > > > distribution
> > > > > > > > > > > > > > > > works well,
but trying to do this myself so I
> > can
> > > > > learn
> > > > > > > > > what's
> > > > > > > > > > > > > > happening
> > > > > > > > > > > > > > > > behind the
scene.)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have a Master
& a Slave.  When I start HDFS
> > on
> > > > the
> > > > > > > > master,
> > > > > > > > > I
> > > > > > > > > > > get
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > saying "10.xxx.xxx.xxx
(Permission denied)" -
> > > where
> > > > > > > 10.xxx
> > > > > > > > is
> > > > > > > > > > IP
> > > > > > > > > > > > > > address
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the slave.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Basic problem
(I think) is that I can't ssh
> > from
> > > > the
> > > > > > > master
> > > > > > > > > EC2
> > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the Slave
EC2 instance.  What's the best way
> to
> > > fix
> > > > > it?
> > > > > > >  I
> > > > > > > > > > think
> > > > > > > > > > > I
> > > > > > > > > > > > > need
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > "Key Pair"
file on my master.  I have a key
> > pair
> > > on
> > > > > my
> > > > > > > > local
> > > > > > > > > > > > machine,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > how do I transfer
it to the EC2 machine?  (I
> > > know,
> > > > I
> > > > > > > know,
> > > > > > > > I
> > > > > > > > > > > > agree..
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > dumb :)  Should
I FTP it?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Please help.
 Thanks.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message