hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Wilcox <Tom.Wil...@cse.org.uk>
Subject RE: zookeeper connection problem in pseudo mode
Date Fri, 06 Jan 2012 16:01:11 GMT
Yong,

Following Leonardo's response, when I run ifconfig on our regionserver (also our namenode)
I get the following:


[hadoop1@namenode hadoop-1.0.0]$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:15:5D:00:E5:09
          inet addr:10.0.0.235  Bcast:10.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19919141 errors:0 dropped:0 overruns:0 frame:4
          TX packets:12620289 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15386337069 (14.3 GiB)  TX bytes:5571908867 (5.1 GiB)
          Interrupt:9 Base address:0x6000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:3121776 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3121776 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1272206567 (1.1 GiB)  TX bytes:1272206567 (1.1 GiB)

Here's the contents of our hbase-site.xml, however, please note that we are running our zookeepers
outside of hbase (i.e. in $HBASE_HOME/conf/hbase-env.sh we have set the following:  "export
HBASE_MANAGES_ZK=false" ) and we are running in fully distributed mode, so I am not sure how
useful this will be to you:

[hadoop1@namenode hadoop-1.0.0]$ cat /opt/hbase/hbase-0.92/conf/hbase-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
 * Copyright 2010 The Apache Software Foundation
 *
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
-->
<configuration>
<!-- Fully Distributed -->
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://namenode:8020/hbase</value>
    <description>The directory shared by RegionServers.</description>
  </property>
  <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
    <description>
                The mode the cluster will be in. Possible values are
                false: standalone and pseudo-distributed setups with managed Zookeeper
                true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
    </description>
  </property>

<!-- ZooKeeper config -->
    <property>
      <name>hbase.zookeeper.property.clientPort</name>
      <value>2181</value>
      <description>
                Property from ZooKeeper's config zoo.cfg.
                The port at which the clients will connect.
      </description>
    </property>
    <property>
      <name>hbase.zookeeper.quorum</name>
      <value>namenode, jobtracker, slave0</value>
      <description>
        Comma separated list of servers in the ZooKeeper Quorum.
        For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".
        By default this is set to localhost for local and pseudo-distributed modes
        of operation. For a fully-distributed setup, this should be set to a full
        list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh
        this is the list of servers which we will start/stop ZooKeeper on.
      </description>
    </property>
    <property>
      <name>hbase.zookeeper.property.dataDir</name>
      <value>/zookeeper_data</value>
      <description>
                Property from ZooKeeper's config zoo.cfg.
                The directory where the snapshot is stored.
      </description>
    </property>

<!-- Pseudo-distributed -->
<!--
        <property>
                <name>hbase.rootdir</name>
                <value>hdfs://namenode:8020/hbase</value>
                <description>
                        The directory shared by RegionServers.
                </description>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
                <description>
                        The replication count for HLog and HFile storage. Should not be greater
than                                                                                     
                      HDFS datanode count.
                </description>
        </property>
-->
<!-- Standalone -->
<!--
        <property>
            <name>hbase.rootdir</name>
            <value>file:///hbase_data</value>
        </property>
-->
</configuration>



It does sound like it could be a ZooKeeper issue. Have you tried installing zookeeper separately
as we have done? Note that the pseudo-distributed settings we used are still commented out
at the bottom.

HTH

Tom

________________________________________
From: Leonardo Gamas [leogamas@jusbrasil.com.br]
Sent: 06 January 2012 15:05
To: user@hbase.apache.org
Subject: Re: zookeeper connection problem in pseudo mode

Seems you does not have a loopback device configured. How many network
interfaces appear when you run the command:

$ ifconfig


2012/1/6 yonghu <yongyong313@gmail.com>

> Hallo,
>
> I input the first command
> $ telnet localhost 2181
> the output is
>
> Trying 127.0.0.1...
> Trying ::1...
> telnet: Unable to connect to remote host: Network is unreachable
>
> As I run the pseudo mode, server and client are the same. So I input
>
>  netstat -na | grep LISTEN | grep 2181
>
> But I got no output information.
>
> Yong
> On Fri, Jan 6, 2012 at 2:25 PM, Leonardo Gamas
> <leogamas@jusbrasil.com.br> wrote:
> > Yong,
> >
> > Just to have more info, if you run locally to your server:
> >
> > $ telnet localhost 2181
> >
> > It connects?
> >
> > And if you run this command from your host to the server:
> >
> > $ telnet <server_ip or hostname> 2181
> >
> > What happens?
> >
> > In the server:
> >
> > $ netstat -na | grep LISTEN | grep 2181
> >
> > What is printed?
> >
> > 2012/1/6 yonghu <yongyong313@gmail.com>
> >
> >> Thanks for your response. I use Ubuntu 10.04 and the following is the
> >> command information I got
> >>
> >> huyong@huyong-laptop:~$ sudo /etc/init.d/iptables stop
> >> [sudo] password for huyong:
> >> sudo: /etc/init.d/iptables: command not found
> >>
> >> The way you did can't work for me. The way I used is from
> >> https://help.ubuntu.com/community/IptablesHowTo
> >>
> >> Can you show me your hbase configuration. I mean the content of
> >> hbase-site.xml. Or you can give me link that I can check if I did the
> >> right configuration steps. Thanks very much.
> >>
> >> Yong
> >>
> >> On Fri, Jan 6, 2012 at 12:23 PM, Tom Wilcox <Tom.Wilcox@cse.org.uk>
> wrote:
> >> > Hi Yong,
> >> >
> >> > When I run:
> >> >
> >> > sudo /etc/init.d/iptables stop
> >> >
> >> > I see the following output:
> >> >
> >> > iptables: Flushing firewall rules:                         [  OK  ]
> >> > iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
> >> > iptables: Unloading modules:                               [  OK  ]
> >> >
> >> > And my connection stopped being refused. Since I wasn't 100% sure what
> >> ports needed opening for HBase and co to work and my system was on a
> closed
> >> development network, I was happy just to stop the firewall services
> >> completely as above, rather than to modify the rules.
> >> >
> >> > We have successfully combined (Hadoop 0.20.2 with HBase 0.90.4) and
> >> (Hadoop 1.0.0 with HBase 0.92).
> >> >
> >> > HTH
> >> >
> >> > Tom
> >> > ________________________________________
> >> > From: yonghu [yongyong313@gmail.com]
> >> > Sent: 05 January 2012 21:22
> >> > To: user@hbase.apache.org
> >> > Subject: Re: zookeeper connection problem in pseudo mode
> >> >
> >> > this is how I stop the firewall.
> >> https://help.ubuntu.com/community/IptablesHowTo
> >> >
> >> > sudo nano -w /root/fw.stop
> >> >
> >> > echo "Stopping firewall and allowing everyone..."
> >> > iptables -F
> >> > iptables -X
> >> > iptables -t nat -F
> >> > iptables -t nat -X
> >> > iptables -t mangle -F
> >> > iptables -t mangle -X
> >> > iptables -P INPUT ACCEPT
> >> > iptables -P FORWARD ACCEPT
> >> > iptables -P OUTPUT ACCEPT
> >> >
> >> > sudo /root/fw.stop
> >> >
> >> > at last I saw the output:
> >> > huyong@huyong-laptop:~/hbase-0.90.3/bin$ sudo /root/fw.stop
> >> > stopping firewall
> >> >
> >> > I am not sure weather it is what you mean to close the firewall. I
> >> > don't know if my Hadoop version 0.20.2 and Hbase 0.90.3 version are
> >> > compatible.
> >> > Yong
> >> >
> >> >
> >> > On Thu, Jan 5, 2012 at 6:32 PM, Royston Sellman
> >> > <royston.sellman@googlemail.com> wrote:
> >> >> Just to check - did you disable it the way Tom suggested? By stopping
> >> iptables?
> >> >> It's not sufficient just to turn off firewall from the control
> >> panel/app.
> >> >>
> >> >> Best,
> >> >> Royston
> >> >>
> >> >>
> >> >> On 5 Jan 2012, at 17:00, yonghu wrote:
> >> >>
> >> >>> I have already disabled ipv6 and close the firewall, but I still
get
> >> >>> the same problem. :(
> >> >>>
> >> >>> Yong
> >> >>>
> >> >>> On Thu, Jan 5, 2012 at 5:16 PM, yonghu <yongyong313@gmail.com>
> wrote:
> >> >>>> I set up a pseudo mode, it also needs to close the firewall?
> >> >>>>
> >> >>>> Yong
> >> >>>>
> >> >>>> On Thu, Jan 5, 2012 at 4:59 PM, Tom Wilcox <Tom.Wilcox@cse.org.uk>
> >> wrote:
> >> >>>>> I've seen that error before. Have you tried turning off
your
> >> firewalls on each node?
> >> >>>>>
> >> >>>>> I do it like this:
> >> >>>>>
> >> >>>>> cat $HADOOP_INSTALL/conf/slaves | xargs -I{} ssh root@{}
> >> '/etc/init.d/iptables stop'
> >> >>>>> ssh root@jobtracker '/etc/init.d/iptables stop'
> >> >>>>> ssh root@namenode '/etc/init.d/iptables stop'
> >> >>>>>
> >> >>>>> Worth a crack if you haven't tried it yet.
> >> >>>>> ________________________________________
> >> >>>>> From: Leonardo Gamas [leogamas@jusbrasil.com.br]
> >> >>>>> Sent: 05 January 2012 15:45
> >> >>>>> To: user@hbase.apache.org
> >> >>>>> Subject: Re: zookeeper connection problem in pseudo mode
> >> >>>>>
> >> >>>>> I had a similar problem some time ago. The problem seems
to be in
> >> the ipv6
> >> >>>>> configuration. I solved it disabling ipv6 on my host editing
> >> /etc/hosts.
> >> >>>>>
> >> >>>>> 2012/1/5 yonghu <yongyong313@gmail.com>
> >> >>>>>
> >> >>>>>> Hallo,
> >> >>>>>>
> >> >>>>>> I tried the pseudo distribution mode of HBase. The
Hadoop
> version is
> >> >>>>>> 0.20.2. I configured the hbase-site.xml as
> >> >>>>>>
> >> >>>>>> <configuration>
> >> >>>>>>  <property>
> >> >>>>>>    <name>hbase.rootdir</name>
> >> >>>>>>    <value>hdfs://localhost:9000/hbase</value>
> >> >>>>>>  </property>
> >> >>>>>>  <property>
> >> >>>>>>    <name>dfs.replication</name>
> >> >>>>>>    <value>1</value>
> >> >>>>>>  </property>
> >> >>>>>> </configuration>
> >> >>>>>>
> >> >>>>>> as same as in core-site.xml and hdfs-site.xml files
of Hadoop,
> >> separately.
> >> >>>>>>
> >> >>>>>> I can successfully start HBase. But when I tried to
create a
> table,
> >> I
> >> >>>>>> got following error information:
> >> >>>>>> ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException:
> HBase
> >> is
> >> >>>>>> able to connect to ZooKeeper but the connection closes
> immediately.
> >> >>>>>> This could be a sign that the server has too many connections
> (30 is
> >> >>>>>> the default). Consider inspecting your ZK server logs
for that
> error
> >> >>>>>> and then make sure you are reusing HBaseConfiguration
as often as
> >> you
> >> >>>>>> can. See HTable's javadoc for more information.
> >> >>>>>>
> >> >>>>>> And I checked my ZK log and found:
> >> >>>>>>
> >> >>>>>> 2012-01-05 15:59:02,909 INFO org.apache.zookeeper.ClientCnxn:
> >> Opening
> >> >>>>>> socket connection to server localhost/0:0:0:0:0:0:0:1:2181
> >> >>>>>> 2012-01-05 15:59:02,909 WARN org.apache.zookeeper.ClientCnxn:
> >> Session
> >> >>>>>> 0x134ae4d4a6d0001 for server null, unexpected error,
closing
> socket
> >> >>>>>> connection and attempting reconnect
> >> >>>>>> java.net.ConnectException: Connection refused
> >> >>>>>>        at sun.nio.ch.SocketChannelImpl.checkConnect(Native
> Method)
> >> >>>>>>        at
> >> >>>>>>
> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> >> >>>>>>        at
> >> >>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
> >> >>>>>>
> >> >>>>>> This error information appeared repeatly. Can anyone
tell how to
> >> solve
> >> >>>>>> this problem?
> >> >>>>>>
> >> >>>>>> Thanks!
> >> >>>>>>
> >> >>>>>> Yong
> >> >>>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> --
> >> >>>>>
> >> >>>>> *Leonardo Gamas*
> >> >>>>> Software Engineer
> >> >>>>> +557134943514
> >> >>>>> +557581347440
> >> >>>>> leogamas@jusbrasil.com.br
> >> >>>>> www.jusbrasil.com.br
> >> >>
> >>
> >
> >
> >
> > --
> >
> > *Leonardo Gamas*
> > Software Engineer
> > T +55 (71) 3494-3514
> > C +55 (75) 8134-7440
> > leogamas@jusbrasil.com.br
> > www.jusbrasil.com.br
>



--

*Leonardo Gamas*
Software Engineer
T +55 (71) 3494-3514
C +55 (75) 8134-7440
leogamas@jusbrasil.com.br
www.jusbrasil.com.br

Mime
View raw message