hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vaibhav Puranik <vpura...@gmail.com>
Subject Re: Starting HBase in fully distributed mode...
Date Mon, 07 Dec 2009 21:45:22 GMT
Or if you have all of your instances in default group. YOu can add default
group into default group.

Regards,
Vaibhav

On Mon, Dec 7, 2009 at 1:43 PM, Vaibhav Puranik <vpuranik@gmail.com> wrote:

> Changing connection method to custom might be ok. I don't remember it
> exactly.
> Unfortunately, there is no way to add an instance to a security group once
> it's booted. You have to specify the security group at launch.
> (check this FAQ from Amazon -
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1145#13
> )
>
> You will have to shutdown your cluster. Reboot the instances with 'hbase'
> security group.
>
> Regards,
> Vaibhav
>
>
>
> On Mon, Dec 7, 2009 at 12:05 PM, Something Something <
> mailinglists19@gmail.com> wrote:
>
>> After doing that.. "Connection Method" was changed automatically to
>> "Custom...".  Is that correct?
>>
>> Next step is... "Add all the machines on which hbase is running - master
>> and
>> slaves to hbase
>> group"
>>
>> Does that mean -
>>
>> Connection Method: SSH
>> Protocol: TCP
>> From & To Ports: ?
>> Source:  Master IP
>>
>> Connection Method: SSH
>> Protocol: TCP
>> From & To Ports: ?
>> Source:  Slave 1 IP
>>
>> Connection Method: SSH
>> Protocol: TCP
>> From & To Ports: ?
>> Source:  Slave 2 IP
>>
>> Is that what you mean?  Please let me know.  Thanks again for your help.
>>
>>
>> On Mon, Dec 7, 2009 at 11:34 AM, Vaibhav Puranik <vpuranik@gmail.com>
>> wrote:
>>
>> > Select the following fields:
>> >
>> > Connection Method:All
>> > Leave Protocal, From Port and To Port empty (or default)
>> > Type 'hbase' (or the same group name) in the source field. Notice that
>> the
>> > source field says IP or Group. You can type any group name there.
>> >
>> > Regards,
>> > Vaibhav Puranik
>> > Gumgum
>> >
>> > On Mon, Dec 7, 2009 at 11:30 AM, Something Something <
>> > mailinglists19@gmail.com> wrote:
>> >
>> > > Hmm..  not sure what you mean by "Add hbase into hbase"
>> > >
>> > > I added security group 'hbase' using AWS Console.  The screen has
>> > following
>> > > columns at the bottom:
>> > >
>> > > Connection Method, Protocol, From Port, To Port, Source, Actions
>> > >
>> > > Please let me know.  Thanks.
>> > >
>> > >
>> > > On Mon, Dec 7, 2009 at 11:12 AM, Vaibhav Puranik <vpuranik@gmail.com>
>> > > wrote:
>> > >
>> > > > Here is what I suggest:
>> > > >
>> > > > Make a security group - say hbase.
>> > > > Add hbase into hbase.
>> > > >
>> > > > Add all the machines on which hbase is running - master and slaves
>> to
>> > > hbase
>> > > > group.
>> > > >
>> > > > And use private names that start with domU-XXXXXXXXXXXX in
>> > configuration
>> > > > files.
>> > > >
>> > > > This should work.
>> > > >
>> > > > Regards,
>> > > > Vaibhav
>> > > >
>> > > > On Sun, Dec 6, 2009 at 9:04 PM, Something Something <
>> > > > mailinglists19@gmail.com> wrote:
>> > > >
>> > > > > After using internal IPs on EC2, Hadoop started cleanly, with no
>> > errors
>> > > > in
>> > > > > any of the 4 logs (on Master) & 2 logs (on each Slave).
>> > > > >
>> > > > > But when I start HBase, I get this...
>> > > > >
>> > > > > java.net.ConnectException: Connection refused
>> > > > >         at sun.nio.ch.Net.connect(Native Method)
>> > > > >        at
>> > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
>> > > > >        at
>> > java.nio.channels.SocketChannel.open(SocketChannel.java:146)
>> > > > >        at
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:323)
>> > > > >        at
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:356)
>> > > > >        at
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:603)
>> > > > >        at
>> > > > >
>> > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:488)
>> > > > > 2009-12-07 04:24:56,006 INFO
>> > > > > org.apache.zookeeper.server.quorum.FastLeaderElection:
>> Notification
>> > > time
>> > > > > out: 400
>> > > > > 2009-12-07 04:24:56,428 WARN
>> > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open
>> > > channel
>> > > > to
>> > > > > 2 at election address domU-12-31-38-00-44-99/10.252.75.133:3888
>> > > > > java.net.ConnectException: Connection refused
>> > > > >         at sun.nio.ch.Net.connect(Native Method)
>> > > > >        at
>> > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
>> > > > >        at
>> > java.nio.channels.SocketChannel.open(SocketChannel.java:146)
>> > > > >        at
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:323)
>> > > > >        at
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:356)
>> > > > >        at
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:603)
>> > > > >        at
>> > > > >
>> > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:488)
>> > > > > 2009-12-07 04:24:56,434 WARN
>> > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open
>> > > channel
>> > > > to
>> > > > > 1 at election address domU-12-31-38-00-91-99/10.252.150.226:3888
>> > > > >
>> > > > >
>> > > > >
>> > > > > At first I thought this was because Port 3888 is not open, so I
>> added
>> > > > > '3888'
>> > > > > to my "default group" under "Security Groups" of EC2 Console with
>> > > source
>> > > > > set
>> > > > > to 0.0.0.0/0.  To the best of my knowledge that's the way to open
>> a
>> > > port
>> > > > > under EC2 (correct?)
>> > > > >
>> > > > > Later I created 3 new EC2 instances from scratch, but still got
>> these
>> > > > > messages.  Seems like ZooKeeper is not starting automatically on
>> this
>> > > > port
>> > > > > on the Slaves.  Any reason why?  Please help.  Thanks.
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Fri, Dec 4, 2009 at 3:03 PM, Patrick Hunt <phunt@apache.org>
>> > wrote:
>> > > > >
>> > > > > > That is weird because it works for me. I just tried your example
>> > > (eth0
>> > > > vs
>> > > > > > ath0) and I was able to "echo stat |nc <ip eth0|ath0> 2181" as
>> well
>> > > as
>> > > > > > connect a ZK client successfully using either IP address.
>> > > > > >
>> > > > > > netstat -a shows this:
>> > > > > > tcp6       0      0 [::]:2181   [::]:*    LISTEN
>> > > > > >
>> > > > > >
>> > > > > > What do you see for netstat?
>> > > > > >
>> > > > > > I'm on ipv4, are you running ipv6?
>> > > > > >
>> > > > > >
>> > > > > > Patrick
>> > > > > >
>> > > > > >
>> > > > > > Jean-Daniel Cryans wrote:
>> > > > > >
>> > > > > >> It seems not... For example on my dev machine I have an
>> interface
>> > > for
>> > > > > >> wired network and another one for wireless. When I start ZK it
>> > binds
>> > > > > >> on only one interface so if I connect to the other IP it
>> doesn't
>> > > work.
>> > > > > >>
>> > > > > >> J-D
>> > > > > >>
>> > > > > >> On Fri, Dec 4, 2009 at 2:35 PM, Patrick Hunt <phunt@apache.org
>> >
>> > > > wrote:
>> > > > > >>
>> > > > > >>> Sorry, but I'm still not able to grok this issue. Perhaps you
>> can
>> > > > shed
>> > > > > >>> more
>> > > > > >>> light: here's the exact code from our server to bind to the
>> > client
>> > > > > port:
>> > > > > >>>
>> > > > > >>>   ss.socket().bind(new InetSocketAddress(port));
>> > > > > >>>
>> > > > > >>> my understanding from the java docs is this:
>> > > > > >>>
>> > > > > >>>   public InetSocketAddress(int port)
>> > > > > >>>       "Creates a socket address where the IP address is the
>> > > wildcard
>> > > > > >>> address and the port number a specified value."
>> > > > > >>>
>> > > > > >>>
>> > > > > >>> afaik this binds the socket onto the specified port for any ip
>> on
>> > > any
>> > > > > >>> interface of the host. Where am I going wrong?
>> > > > > >>>
>> > > > > >>> Patrick
>> > > > > >>>
>> > > > > >>> Jean-Daniel Cryans wrote:
>> > > > > >>>
>> > > > > >>>> The first two definitions here is what I'm talking about
>> > > > > >>>>
>> > > > > >>>>
>> > > > >
>> > > >
>> > >
>> >
>> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1346
>> > > > > >>>>
>> > > > > >>>> So by default it usually doesn't listen on the interface
>> > > associated
>> > > > > >>>> with the hostname ec2-IP-compute-1.amazonaws.com but on the
>> > other
>> > > > one
>> > > > > >>>> (IIRC starts with dom-).
>> > > > > >>>>
>> > > > > >>>> J-D
>> > > > > >>>>
>> > > > > >>>> On Fri, Dec 4, 2009 at 12:41 PM, Patrick Hunt <
>> phunt@apache.org
>> > >
>> > > > > wrote:
>> > > > > >>>>
>> > > > > >>>>> I'm not familiar with ec2, when you say "listen on private
>> > > > hostname"
>> > > > > >>>>> what
>> > > > > >>>>> does that mean? Do you mean "by default listen on an
>> interface
>> > > with
>> > > > a
>> > > > > >>>>> non-routable (localonly) ip"? Or something else. Is there an
>> > aws
>> > > > page
>> > > > > >>>>> you
>> > > > > >>>>> can point me to?
>> > > > > >>>>>
>> > > > > >>>>> Patrick
>> > > > > >>>>>
>> > > > > >>>>> Jean-Daniel Cryans wrote:
>> > > > > >>>>>
>> > > > > >>>>>> When you saw:
>> > > > > >>>>>>
>> > > > > >>>>>> org.apache.hadoop.hdfs.server.namenode.SafeModeException:
>> > Cannot
>> > > > > >>>>>> delete
>> > > > > >>>>>> /ebs1/mapred/system,/ebs2/mapred/system. Name node is in
>> safe
>> > > > mode.
>> > > > > >>>>>> The ratio of reported blocks 0.0000 has not reached the
>> > > threshold
>> > > > > >>>>>> 0.9990.
>> > > > > >>>>>> *Safe
>> > > > > >>>>>> mode will be turned off automatically*.
>> > > > > >>>>>>
>> > > > > >>>>>> It means that HDFS is blocking everything (aka safe mode)
>> > until
>> > > > all
>> > > > > >>>>>> datanodes reported for duty (and then it waits for 30
>> seconds
>> > to
>> > > > > make
>> > > > > >>>>>> sure).
>> > > > > >>>>>>
>> > > > > >>>>>> When you saw:
>> > > > > >>>>>>
>> > > > > >>>>>> Caused by:
>> > org.apache.zookeeper.KeeperException$NoNodeException:
>> > > > > >>>>>> KeeperErrorCode = *NoNode for /hbase/master*
>> > > > > >>>>>>
>> > > > > >>>>>> It means that the Master node didn't write his znode in
>> > > Zookeeper
>> > > > > >>>>>> because... when you saw:
>> > > > > >>>>>>
>> > > > > >>>>>> 2009-12-04 07:07:37,149 WARN
>> org.apache.zookeeper.ClientCnxn:
>> > > > > >>>>>> Exception
>> > > > > >>>>>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@10e35d5
>> > > > > >>>>>> java.net.ConnectException: Connection refused
>> > > > > >>>>>>
>> > > > > >>>>>> It really means that the connection was refused. It then
>> says
>> > it
>> > > > > >>>>>> attempted to connect to
>> > > > ec2-174-129-127-141.compute-1.amazonaws.com
>> > > > > >>>>>> but wasn't able to. AFAIK in EC2 the java processes tend to
>> > > listen
>> > > > > on
>> > > > > >>>>>> their private hostname not the public one (which would be
>> bad
>> > > > > >>>>>> anyways).
>> > > > > >>>>>>
>> > > > > >>>>>> Bottom line, make sure stuff listens where they are
>> expected
>> > and
>> > > > it
>> > > > > >>>>>> should then work well.
>> > > > > >>>>>>
>> > > > > >>>>>> J-D
>> > > > > >>>>>>
>> > > > > >>>>>> On Fri, Dec 4, 2009 at 11:23 AM, Something Something
>> > > > > >>>>>> <mailinglists19@gmail.com> wrote:
>> > > > > >>>>>>
>> > > > > >>>>>>> Hadoop: 0.20.1
>> > > > > >>>>>>>
>> > > > > >>>>>>> HBase: 0.20.2
>> > > > > >>>>>>>
>> > > > > >>>>>>> Zookeeper: The one which gets started by default by HBase.
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>> HBase logs:
>> > > > > >>>>>>>
>> > > > > >>>>>>> 1)  Master log shows this WARN message, but then it says
>> > > > > 'connection
>> > > > > >>>>>>> successful'
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>> 2009-12-04 07:07:37,149 WARN
>> org.apache.zookeeper.ClientCnxn:
>> > > > > >>>>>>> Exception
>> > > > > >>>>>>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@10e35d5
>> > > > > >>>>>>> java.net.ConnectException: Connection refused
>> > > > > >>>>>>>     at sun.nio.ch.SocketChannelImpl.checkConnect(Native
>> > Method)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
>> > > > > >>>>>>> 2009-12-04 07:07:37,150 WARN
>> org.apache.zookeeper.ClientCnxn:
>> > > > > >>>>>>> Ignoring
>> > > > > >>>>>>> exception during shutdown input
>> > > > > >>>>>>> java.nio.channels.ClosedChannelException
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>> > > > > >>>>>>>     at
>> > > > > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> > > > > >>>>>>> 2009-12-04 07:07:37,150 WARN
>> org.apache.zookeeper.ClientCnxn:
>> > > > > >>>>>>> Ignoring
>> > > > > >>>>>>> exception during shutdown output
>> > > > > >>>>>>> java.nio.channels.ClosedChannelException
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > >
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> > > > > >>>>>>> 2009-12-04 07:07:37,199 INFO
>> > > > > >>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>> > > > > >>>>>>> -ROOT- region unset (but not set to be reassigned)
>> > > > > >>>>>>> 2009-12-04 07:07:37,200 INFO
>> > > > > >>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>> > > > > >>>>>>> ROOT inserted into regionsInTransition
>> > > > > >>>>>>> 2009-12-04 07:07:37,667 INFO
>> org.apache.zookeeper.ClientCnxn:
>> > > > > >>>>>>> Attempting
>> > > > > >>>>>>> connection to server
>> > > > > >>>>>>>
>> > ec2-174-129-127-141.compute-1.amazonaws.com/10.252.146.65:2181
>> > > > > >>>>>>> 2009-12-04 07:07:37,668 INFO
>> org.apache.zookeeper.ClientCnxn:
>> > > > > Priming
>> > > > > >>>>>>> connection to java.nio.channels.SocketChannel[connected
>> > local=/
>> > > > > >>>>>>> 10.252.162.19:46195 remote=
>> > > > > >>>>>>>
>> > ec2-174-129-127-141.compute-1.amazonaws.com/10.252.146.65:2181
>> > > ]
>> > > > > >>>>>>> 2009-12-04 07:07:37,670 INFO
>> org.apache.zookeeper.ClientCnxn:
>> > > > > Server
>> > > > > >>>>>>> connection successful
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>> 2)  Regionserver log shows this... but later seems to have
>> > > > > recovered:
>> > > > > >>>>>>>
>> > > > > >>>>>>> 2009-12-04 07:07:36,576 WARN
>> org.apache.zookeeper.ClientCnxn:
>> > > > > >>>>>>> Exception
>> > > > > >>>>>>> closing session 0x0 to sun.nio.ch.SelectionKeyImpl@4ee70b
>> > > > > >>>>>>> java.net.ConnectException: Connection refused
>> > > > > >>>>>>>     at sun.nio.ch.SocketChannelImpl.checkConnect(Native
>> > Method)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
>> > > > > >>>>>>> 2009-12-04 07:07:36,611 WARN
>> org.apache.zookeeper.ClientCnxn:
>> > > > > >>>>>>> Ignoring
>> > > > > >>>>>>> exception during shutdown input
>> > > > > >>>>>>> java.nio.channels.ClosedChannelException
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>> > > > > >>>>>>>     at
>> > > > > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> > > > > >>>>>>> 2009-12-04 07:07:36,611 WARN
>> org.apache.zookeeper.ClientCnxn:
>> > > > > >>>>>>> Ignoring
>> > > > > >>>>>>> exception during shutdown output
>> > > > > >>>>>>> java.nio.channels.ClosedChannelException
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > >
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
>> > > > > >>>>>>> 2009-12-04 07:07:36,742 WARN
>> > > > > >>>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed
>> to
>> > > set
>> > > > > >>>>>>> watcher
>> > > > > >>>>>>> on
>> > > > > >>>>>>> ZNode /hbase/master
>> > > > > >>>>>>>
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > > > > >>>>>>> KeeperErrorCode = ConnectionLoss for /hbase/master
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > >
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > >
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>> > > > > >>>>>>>     at
>> > > org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.watchMasterAddress(ZooKeeperWrapper.java:304)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.watchMasterAddress(HRegionServer.java:385)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.reinitializeZooKeeper(HRegionServer.java:315)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.reinitialize(HRegionServer.java:306)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:276)
>> > > > > >>>>>>>     at
>> > > > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> > > > > >>>>>>> Method)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > > >
>> > >
>> >
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > > >
>> > >
>> >
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.doMain(HRegionServer.java:2474)
>> > > > > >>>>>>>     at
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2542)
>> > > > > >>>>>>> 2009-12-04 07:07:36,743 WARN
>> > > > > >>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable
>> to
>> > > set
>> > > > > >>>>>>> watcher
>> > > > > >>>>>>> on
>> > > > > >>>>>>> ZooKeeper master address. Retrying.
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>> 3)  Zookeepr log:  Nothing much in there... just a
>> starting
>> > > > message
>> > > > > >>>>>>> line..
>> > > > > >>>>>>> followed by
>> > > > > >>>>>>>
>> > > > > >>>>>>> ulimit -n 1024
>> > > > > >>>>>>>
>> > > > > >>>>>>> I looked at archives.  There was one mail that talked
>> about
>> > > > > 'ulimit'.
>> > > > > >>>>>>>  Wonder if that has something to do with it.
>> > > > > >>>>>>>
>> > > > > >>>>>>> Thanks for your help.
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>> On Fri, Dec 4, 2009 at 8:18 AM, Mark Vigeant
>> > > > > >>>>>>> <mark.vigeant@riskmetrics.com>wrote:
>> > > > > >>>>>>>
>> > > > > >>>>>>>  When I first started my hbase cluster, it too gave me the
>> > > nonode
>> > > > > for
>> > > > > >>>>>>>> /hbase/master several times before it started working,
>> and I
>> > > > > believe
>> > > > > >>>>>>>> this is
>> > > > > >>>>>>>> a common beginner's error (I've seen it in a few emails
>> in
>> > the
>> > > > > past
>> > > > > >>>>>>>> 2
>> > > > > >>>>>>>> weeks).
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> What versions of HBase, Hadoop and ZooKeeper are you
>> using?
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> Also, take a look in your HBASE_HOME/logs folder. That
>> would
>> > > be
>> > > > a
>> > > > > >>>>>>>> good
>> > > > > >>>>>>>> place to start looking for some answers.
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> -Mark
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> -----Original Message-----
>> > > > > >>>>>>>> From: Something Something [mailto:
>> mailinglists19@gmail.com]
>> > > > > >>>>>>>> Sent: Friday, December 04, 2009 2:28 AM
>> > > > > >>>>>>>> To: hbase-user@hadoop.apache.org
>> > > > > >>>>>>>> Subject: Starting HBase in fully distributed mode...
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> Hello,
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> I am trying to get Hadoop/HBase up and running in a fully
>> > > > > >>>>>>>> distributed
>> > > > > >>>>>>>> mode.
>> > > > > >>>>>>>>  For now, I have only *1 Master & 2 Slaves*.
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> The Hadoop starts correctly.. I think.  The only
>> exception I
>> > > see
>> > > > > in
>> > > > > >>>>>>>> various
>> > > > > >>>>>>>> log files is this one...
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> org.apache.hadoop.ipc.RemoteException:
>> > > > > >>>>>>>> org.apache.hadoop.hdfs.server.namenode.SafeModeException:
>> > > Cannot
>> > > > > >>>>>>>> delete
>> > > > > >>>>>>>> /ebs1/mapred/system,/ebs2/mapred/system. Name node is in
>> > safe
>> > > > > mode.
>> > > > > >>>>>>>> The ratio of reported blocks 0.0000 has not reached the
>> > > > threshold
>> > > > > >>>>>>>> 0.9990.
>> > > > > >>>>>>>> *Safe
>> > > > > >>>>>>>> mode will be turned off automatically*.
>> > > > > >>>>>>>>     at
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1696)
>> > > > > >>>>>>>>     at
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1676)
>> > > > > >>>>>>>>     at
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > >
>> > >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
>> > > > > >>>>>>>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> > > > Method)
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> Somehow this doesn't sound critical, so I assumed
>> everything
>> > > was
>> > > > > >>>>>>>> good
>> > > > > >>>>>>>> to
>> > > > > >>>>>>>> go
>> > > > > >>>>>>>> with Hadoop.
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> So then I started HBase and opened a shell (hbase shell).
>> >  So
>> > > > far
>> > > > > >>>>>>>> everything
>> > > > > >>>>>>>> looks good.  Now when I try to run a 'list' command, I
>> keep
>> > > > > getting
>> > > > > >>>>>>>> this
>> > > > > >>>>>>>> message:
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> Caused by:
>> > > org.apache.zookeeper.KeeperException$NoNodeException:
>> > > > > >>>>>>>> KeeperErrorCode = *NoNode for /hbase/master*
>> > > > > >>>>>>>> at
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > >
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>> > > > > >>>>>>>> at
>> > > > > >>>>>>>>
>> > > > >
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>> > > > > >>>>>>>> at
>> > org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
>> > > > > >>>>>>>> at
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:328)
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> Here's what I have in my *Master hbase-site.xml*
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> <configuration>
>> > > > > >>>>>>>>  <property>
>> > > > > >>>>>>>>  <name>hbase.rootdir</name>
>> > > > > >>>>>>>>  <value>hdfs://master:54310/hbase</value>
>> > > > > >>>>>>>>  </property>
>> > > > > >>>>>>>>  <property>
>> > > > > >>>>>>>>  <name>hbase.cluster.distributed</name>
>> > > > > >>>>>>>>  <value>true</value>
>> > > > > >>>>>>>>  </property>
>> > > > > >>>>>>>>  <property>
>> > > > > >>>>>>>>  <name>hbase.zookeeper.property.clientPort</name>
>> > > > > >>>>>>>>  <value>2181</value>
>> > > > > >>>>>>>>  </property>
>> > > > > >>>>>>>>  <property>
>> > > > > >>>>>>>>  <name>hbase.zookeeper.quorum</name>
>> > > > > >>>>>>>>  <value>master,slave1,slave2</value>
>> > > > > >>>>>>>>  </property>
>> > > > > >>>>>>>> <property>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> The *Slave *hbase-site.xml are set as follows:
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>  <property>
>> > > > > >>>>>>>>  <name>hbase.rootdir</name>
>> > > > > >>>>>>>>  <value>hdfs://master:54310/hbase</value>
>> > > > > >>>>>>>>  </property>
>> > > > > >>>>>>>>  <property>
>> > > > > >>>>>>>>  <name>hbase.cluster.distributed</name>
>> > > > > >>>>>>>>  <value>false</value>
>> > > > > >>>>>>>>  </property>
>> > > > > >>>>>>>>  <property>
>> > > > > >>>>>>>>  <name>hbase.zookeeper.property.clientPort</name>
>> > > > > >>>>>>>>  <value>2181</value>
>> > > > > >>>>>>>>  </property>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> In the hbase-env.sh file on ALL 3 machines I have set the
>> > > > > JAVA_HOME
>> > > > > >>>>>>>> and
>> > > > > >>>>>>>> set
>> > > > > >>>>>>>> the HBase classpath as follows:
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> export
>> > > HBASE_CLASSPATH=$HBASE_CLASSPATH:/ebs1/hadoop-0.20.1/conf
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> On *Master* I have added Master & Slaves IP hostnames to
>> > > > > >>>>>>>> *regionservers*
>> > > > > >>>>>>>> file.
>> > > > > >>>>>>>>  On *slaves*, the regionservers file is empty.
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> I have run hadoop namenode -format multiple times, but
>> still
>> > > > keep
>> > > > > >>>>>>>> getting..
>> > > > > >>>>>>>> "NoNode for /hbase/master".  What step did I miss?
>>  Thanks
>> > for
>> > > > > your
>> > > > > >>>>>>>> help.
>> > > > > >>>>>>>>
>> > > > > >>>>>>>> This email message and any attachments are for the sole
>> use
>> > of
>> > > > the
>> > > > > >>>>>>>> intended
>> > > > > >>>>>>>> recipients and may contain proprietary and/or
>> confidential
>> > > > > >>>>>>>> information
>> > > > > >>>>>>>> which
>> > > > > >>>>>>>> may be privileged or otherwise protected from disclosure.
>> > Any
>> > > > > >>>>>>>> unauthorized
>> > > > > >>>>>>>> review, use, disclosure or distribution is prohibited. If
>> > you
>> > > > are
>> > > > > >>>>>>>> not
>> > > > > >>>>>>>> an
>> > > > > >>>>>>>> intended recipient, please contact the sender by reply
>> email
>> > > and
>> > > > > >>>>>>>> destroy
>> > > > > >>>>>>>> the
>> > > > > >>>>>>>> original message and any copies of the message as well as
>> > any
>> > > > > >>>>>>>> attachments to
>> > > > > >>>>>>>> the original message.
>> > > > > >>>>>>>>
>> > > > > >>>>>>>>
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message