hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Starting HBase in fully distributed mode...
Date Fri, 11 Dec 2009 08:06:10 GMT
> ./bin/hbase-ec2 login testcluster
> 
> Use this to login.  I tried running this from my local machine, but nothing
> *noteworthy* happened.

Did you replace "testcluster" with the name you used when launching your
cluster, assuming they are different? The scripts address clusters by the
labels you give them when launching them. E.g.

   ./bin/hbase-ec2 launch foo 3 3 

launches a cluster named "foo", and

   ./bin/hbase-ec2 login foo

opens a SSH shell on the master of cluster "foo".

> Did you also create similar scripts for Hadoop?

Hadoop has its own set of EC2 scripts. I used those as the basis for ours.
You can't use the HBase and Hadoop EC2 scripts together however. 

> Later I want to start a MapReduce job on my Hadoop machines that will
> access this HBase cluster.  How would I do that?

Are your Hadoop machines up on EC2 also? 

Running mapreduce jobs on the HBase cluster itself is a work in progress.

   - Andy



________________________________
From: Something Something <mailinglists19@gmail.com>
To: hbase-user@hadoop.apache.org
Sent: Thu, December 10, 2009 8:21:10 PM
Subject: Re: Starting HBase in fully distributed mode...

Andy,

Thanks for the tips.  It's all working now.  I was using a different KeyPair
for EC2_ROOT_SSH_KEY.  Once I changed this to use the root.pem it started
working.  I was able to ssh to the 'master' instance and get into hbase
shell etc.  This script is VERY helpful!  Thank you so much.

A few questions...

1)  The README.txt file says this..

./bin/hbase-ec2 login testcluster

Use this to login.  I tried running this from my local machine, but nothing
*noteworthy* happened.  I wasn't able to get into the hbase shell from my
local machine.  Anyway, this is not a big deal for me.

2)  Did you also create similar scripts for Hadoop?  (I guess I will look
into the trunk!).

3)  Say I use your script to start HBase on a few machines, and start Hadoop
on some other machines.  Later I want to start a MapReduce job on my Hadoop
machines that will access this HBase cluster.  How would I do that?  What
HBase configurations can I use?  So far my Mapreduce job always accesses
HBase on the same machine.

Thanks once again for your help.



On Thu, Dec 10, 2009 at 5:30 PM, Vaibhav Puranik <vpuranik@gmail.com> wrote:

> We have HBase running on EC2 with starting Zookeeper within HBase. We have
> it up since July 2009. No problems so far on Zookeeper front.
>
> Regards,
> Vaibhav Puranik
> Gumgum
>
> On Thu, Dec 10, 2009 at 8:12 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Finally, I was able to get HBase running on EC2 in fully distributed
> mode.
> >  I started ZooKeeper quorum myself and pointed HBase to it.  I was able
> to
> > create tables using HBase shell, ran a Mapreduce job that writes to these
> > tables, and run queries against these tables.  I used HBase shell from
> all
> > 3
> > machines, and they all see the same data confirming that the instances
> are
> > indeed working together.
> >
> > It seems like under EC2, starting ZooKeeper within HBase doesn't work,
> but
> > I
> > could be wrong.
> >
> > In any case, Andrew, I would like to get your scripts working in my
> > environment because without your scripts I don't know how I would grow my
> > cluster from 3 instances to say, 30 :)
> >
> > Thank you so much everyone for your help and for sticking with me.
> >
> >
> > On Wed, Dec 9, 2009 at 8:25 PM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > When I run:
> > >
> > > hbase-ec2 launch-cluster testcluster 3 3
> > >
> > > I keep getting 'lost connection' messages (See below).  Tried this 4
> > > times.  Please help.  Thanks.
> > >
> > >
> > > -------------------------------------------------------------
> > >
> > > Creating/checking security groups
> > > Security group testcluster-master exists, ok
> > > Security group testcluster exists, ok
> > > Security group testcluster-zookeeper exists, ok
> > > Starting ZooKeeper quorum ensemble.
> > > Starting an AMI with ID ami-b0cb29d9 (arch i386) in group
> > > testcluster-zookeeper
> > > Waiting for instance i-9db6f4f5 to start: .................. Started
> > > ZooKeeper instance i-9db6f4f5 as
> > domU-12-31-38-01-7D-D1.compute-1.internal
> > >     Public DNS name is ec2-174-129-148-5.compute-1.amazonaws.com.
> > > Starting an AMI with ID ami-b0cb29d9 (arch i386) in group
> > > testcluster-zookeeper
> > > Waiting for instance i-2db7f545 to start: ................. Started
> > > ZooKeeper instance i-2db7f545 as
> > domU-12-31-38-01-7D-43.compute-1.internal
> > >     Public DNS name is ec2-174-129-157-122.compute-1.amazonaws.com.
> > > Starting an AMI with ID ami-b0cb29d9 (arch i386) in group
> > > testcluster-zookeeper
> > > Waiting for instance i-afb7f5c7 to start: ......................
> Started
> > > ZooKeeper instance i-afb7f5c7 as
> > domU-12-31-38-01-78-F3.compute-1.internal
> > >     Public DNS name is ec2-174-129-179-14.compute-1.amazonaws.com.
> > > ZooKeeper quorum is
> > >
> >
> domU-12-31-38-01-7D-D1.compute-1.internal,domU-12-31-38-01-7D-43.compute-1.internal,domU-12-31-38-01-78-F3.compute-1.internal.
> > > Initializing the ZooKeeper quorum ensemble.
> > >     ec2-174-129-148-5.compute-1.amazonaws.com
> > > lost connection
> > >     ec2-174-129-157-122.compute-1.amazonaws.com
> > > lost connection
> > >     ec2-174-129-179-14.compute-1.amazonaws.com
> > > lost connection
> > >
> > >
> > >
> > >
> > > On Wed, Dec 9, 2009 at 12:46 AM, Seth Ladd <sethladd@gmail.com> wrote:
> > >
> > >> > Sounds like others have used Andrew's script successfully.  The only
> > >> > difference seems to be that it starts a *dedicated* ZooKeeper
> quorum.
> > >> > Should have listened to Mark when he suggested that 4 days ago :)
> > >> >
> > >> > Anyway, I will try Andrew's script tomorrow.
> > >>
> > >> I can vouch that the scripts in svn trunk work.  Thanks to Andrew for
> > >> his help!  I was able to start a 3 node Zookeeper and 5 node HBase
> > >> cluster on EC2 from just the scripts.
> > >>
> > >> Seth
> > >>
> > >
> > >
> >
>



      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message