hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cosmin Cătălin Sanda <cosmincata...@gmail.com>
Subject Re: HBase Master cannot be contacted by Region Servers
Date Wed, 09 Jul 2014 15:17:50 GMT
Everything indicates that master is opened and listens carefully

hduser@slave-1:~$ nc -zv master 60000
Connection to master 60000 port [tcp/*] succeeded!

hduser@master:~/hbase$ netstat -lntu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp6       0      0 192.168.66.60:60000     :::*                    LISTEN

Indeed, it seems that there are 2 interfaces on the slaves

eth0      Link encap:Ethernet  HWaddr 08:00:27:82:a8:8e
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe82:a88e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:216289 errors:0 dropped:0 overruns:0 frame:0
          TX packets:105336 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:172077644 (172.0 MB)  TX bytes:9847997 (9.8 MB)
eth1      Link encap:Ethernet  HWaddr 08:00:27:8d:67:76
          inet addr:192.168.66.61  Bcast:192.168.66.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe8d:6776/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3414 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5566 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:321369 (321.3 KB)  TX bytes:879752 (879.7 KB)

I have added the slave IPs to the /etc/hosts of each of the slaves

/etc/hosts on all slaves

192.168.66.60   master
192.168.66.73   zookeeper-3
192.168.66.71   zookeeper-1
192.168.66.72   zookeeper-2
192.168.66.61   slave-1
192.168.66.62   slave-2
192.168.66.63   slave-3

I have then restarted HBase.....and it worked :). The RS are now
listed in the UI.

Thank you very much for your assistance.
------------------------------------
Cosmin Catalin SANDA
Software Systems Engineer
Phone: +45.27.30.60.35



On Wed, Jul 9, 2014 at 3:35 PM, Ian Brooks <i.brooks@sensewhere.com> wrote:
> Hi,
>
> Have you checked the master process is listening on port 60000 and that you can open
a connection to that port from the slave? could be firewalled off.
>
> Also the regionserver process appears to be using an ip on a different subnet 10.0.2.15,
are there 2 interfaces on the slave?, if so you may want to add the slaves 192.168.66.61 address
to /etc/hosts
>
> -Ian Brooks
>
> On Wednesday 09 Jul 2014 15:29:44 Cosmin Cătălin Sanda wrote:
>> The port should not be needed if the default settings have not been
>> modified. I also don't see how that could be the problem since the
>> default hbase created structure is visible under hdfs. I will try that
>> nevertheless.
>> ------------------------------------
>> Cosmin Catalin SANDA
>> Software Systems Engineer
>> Phone: +45.27.30.60.35
>>
>>
>>
>> On Wed, Jul 9, 2014 at 3:24 PM, Wellington Chevreuil
>> <wellington.chevreuil@gmail.com> wrote:
>> > Hi,
>> >
>> > Had you already tried to set the port on hbase.rootdir for hbase-site.xml, like
below:
>> >
>> >    <property>
>> >        <name>hbase.rootdir</name>
>> >        <value>hdfs://master:8020/hbase</value>
>> >    </property>
>> >
>> > Cheers.
>> >
>> > On 9 Jul 2014, at 14:18, Cosmin Cătălin Sanda <cosmincatalin@gmail.com>
wrote:
>> >
>> >> I have a Hadoop cluster made of 3 slaves and 1 master on top of which
>> >> there is an HBase cluster with 3 RS and 1 master respectively.
>> >> Additionally there is a Zookeeper ensemble on 3 machines.
>> >>
>> >> The Hadoop cluster is functioning correctly as well as the Zookeeper
>> >> ensemble. However, the HBase cluster fails to initialize correctly.
>> >>
>> >> I start HBase it by running ./bin/start-hbase.sh. This correctly
>> >> starts the HBase Master and the Region Servers. The hbase folder in
>> >> hdfs is set-up correctly.
>> >>
>> >> jps on master
>> >>
>> >> hduser@master:~/hbase$ jps
>> >> 5694 HMaster
>> >> 3934 JobHistoryServer
>> >> 3786 NameNode
>> >> 3873 ResourceManager
>> >> 6025 Jps
>> >>
>> >> jps on slaves
>> >>
>> >> 5737 Jps
>> >> 5499 HRegionServer
>> >> 3736 DataNode
>> >> 3820 NodeManager
>> >>
>> >> However, the HBase master does not register the Region Servers as it
>> >> is also apparent from looking at the logs:
>> >>
>> >> master log
>> >>
>> >> [master:master:60000] master.ServerManager: Waiting for region servers
>> >> count to settle; currently checked in 0, slept for 1511 ms, expecting
>> >> minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of
>> >> 1500 ms.
>> >>
>> >> slave log
>> >>
>> >> [regionserver60020] regionserver.HRegionServer: reportForDuty to
>> >> master=master,60000,1404856451890 with port=60020,
>> >> startcode=1404856453874
>> >> [regionserver60020] regionserver.HRegionServer: error telling master we
are up
>> >> com.google.protobuf.ServiceException:
>> >> org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout
>> >> while waiting for channel to be ready for connect. ch :
>> >> java.nio.channels.SocketChannel[connection-pending
>> >> local=/10.0.2.15:53939 remote=master/192.168.66.60:60000]
>> >>
>> >> Here are the configuration details:
>> >>
>> >> /etc/hosts on master
>> >>
>> >> 192.168.66.63   slave-3 # Data Node and Region Server
>> >> 192.168.66.60   master # Name Node and HBase Master
>> >> 192.168.66.73   zookeeper-3 # Zookeeper node
>> >> 192.168.66.71   zookeeper-1 # Zookeeper node
>> >> 192.168.66.72   zookeeper-2 # Zookeeper node
>> >> 192.168.66.62   slave-2 # Data Node and Region Server
>> >> 192.168.66.61   slave-1 # Data Node and Region Server
>> >>
>> >> /etc/hosts on slave-1
>> >>
>> >> 192.168.66.60   master
>> >> 192.168.66.73   zookeeper-3
>> >> 192.168.66.71   zookeeper-1
>> >> 192.168.66.72   zookeeper-2
>> >>
>> >> hbase-site.xml on ALL cluster nodes
>> >>
>> >> <?xml version="1.0"?>
>> >> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> >> <configuration>
>> >>    <property>
>> >>        <name>hbase.tmp.dir</name>
>> >>        <value>/home/hduser/hbase/tmp</value>
>> >>    </property>
>> >>    <property>
>> >>        <name>hbase.rootdir</name>
>> >>        <value>hdfs://master/hbase</value>
>> >>    </property>
>> >>    <property>
>> >>        <name>hbase.cluster.distributed</name>
>> >>        <value>true</value>
>> >>    </property>
>> >>    <property>
>> >>        <name>hbase.local.dir</name>
>> >>        <value>/home/hduser/hbase/local</value>
>> >>    </property>
>> >>    <property>
>> >>        <name>hbase.master.info.port</name>
>> >>        <value>6010</value>
>> >>    </property>
>> >>    <property>
>> >>        <name>hbase.zookeeper.quorum</name>
>> >>        <value>zookeeper-1,zookeeper-2,zookeeper-3,</value>
>> >>    </property>
>> >> </configuration>
>> >>
>> >> regionservers file on master and slaves
>> >>
>> >> slave-3
>> >> slave-1
>> >> slave-2
>> >>
>> >> hbase-env.sh on master and slaves
>> >>
>> >> export JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::"
>> >> export HBASE_OPTS="-XX:+UseConcMarkSweepGC"
>> >> export HBASE_MANAGES_ZK=false
>> >>
>> >> What am I doing wrong so that the nodes cannot talk to each other?
>> >>
>> >> I am using Hadoop 2.4.0 and HBase 0.98.3 along with Zookeeper 3.4.6 on
>> >> Ubuntu Trusty Tahr x64.
>> >

Mime
View raw message