Return-Path: 
 <user-return-38398-apmail-hbase-user-archive=hbase.apache.org@hbase.apache.org>
X-Original-To: apmail-hbase-user-archive@www.apache.org
Delivered-To: apmail-hbase-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C6F3010C18
	for <apmail-hbase-user-archive@www.apache.org>;
 Fri, 23 Aug 2013 10:37:48 +0000 (UTC)
Received: (qmail 78021 invoked by uid 500); 23 Aug 2013 10:37:46 -0000
Delivered-To: apmail-hbase-user-archive@hbase.apache.org
Received: (qmail 77683 invoked by uid 500); 23 Aug 2013 10:37:44 -0000
Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hbase.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hbase.apache.org>
List-Post: <mailto:user@hbase.apache.org>
List-Id: <user.hbase.apache.org>
Reply-To: user@hbase.apache.org
Delivered-To: mailing list user@hbase.apache.org
Received: (qmail 77675 invoked by uid 99); 23 Aug 2013 10:37:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Aug 2013 10:37:43 +0000
X-ASF-Spam-Status: No, hits=2.8 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLY,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of vamshi2105@gmail.com
 designates 209.85.219.54 as permitted sender)
Received: from [209.85.219.54] (HELO mail-oa0-f54.google.com) (209.85.219.54)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Aug 2013 10:37:38 +0000
Received: by mail-oa0-f54.google.com with SMTP id o6so540930oag.13
        for <user@hbase.apache.org>; Fri, 23 Aug 2013 03:37:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        bh=u9dKl0NKGF53WNw+hCf78z57OdrnuutkO9ExU6/sXjE=;
        b=DAd8wYEDy0zvS9rVOdYn72Q98okhYaE/WSdMGzIbfcmTyWyJM7P4BboXylM0WcYE6c
         l2U6UOFN90DoReMBzJHyCE02fUTynLdCHpfPtYl8Bh9jNxbk9jRgTyeEMyTYXm4C9nud
         s55EjBFd82DhAF5JIAI9ykCScv7exYqo4GQUOaieBlk4Z1o5x8skN2Rp2tiYozfxSnTE
         EDFwguJPprkoMvVBPtnASXCLAXBIWouT51A4UWdF8m/QbgWGL8ZjBuMQkY/p5K6CYc1X
         jeHuG5rlyDYhqp49WSKzgGVxpKVDPerIXaKNkW31RBPQwtEfS47yimkWpN+B0rGfFt8W
         0Qqw==
MIME-Version: 1.0
X-Received: by 10.60.132.178 with SMTP id ov18mr19385670oeb.39.1377254237924;
 Fri, 23 Aug 2013 03:37:17 -0700 (PDT)
Received: by 10.76.125.137 with HTTP; Fri, 23 Aug 2013 03:37:17 -0700 (PDT)
In-Reply-To: 
 <CAPvS-K9+m2kvBx0ri3h9T2_s0rEvB2wtstU+O56XU_EY7wDBSw@mail.gmail.com>
References: 
 <CAPVFMeWdQBm64fanekywRgyBGqcapaGB7333=OFbwOVh4v8iSg@mail.gmail.com>
	<760345F9-C3FD-4207-B5EE-0D1391E27327@gmail.com>
	<CAPvS-K9rirZ-96uBYwLzdxbpqs2vMkVchEdW+5zfhxFKXvM9gA@mail.gmail.com>
	<CAPvS-K_uk3qpH3NGE8TyZR4u9NF0PDFg0AgHD8Spgbo3O2oE5A@mail.gmail.com>
	<CAPvS-K9+m2kvBx0ri3h9T2_s0rEvB2wtstU+O56XU_EY7wDBSw@mail.gmail.com>
Date: Fri, 23 Aug 2013 16:07:17 +0530
Message-ID: 
 <CAPVFMeWrED1s0XuE8iLCcwCZqtVrNJAd2EDXZ3f8m2LuaiTx=Q@mail.gmail.com>
Subject: Re: Hbase region server is not communicating with zookeeper and
 stopping after some time it was started
From: Vamshi Krishna <vamshi2105@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=047d7b47215e9df61604e49afdc3
X-Virus-Checked: Checked by ClamAV on apache.org

--047d7b47215e9df61604e49afdc3
Content-Type: text/plain; charset=ISO-8859-1

Hey, I got my problem solved.
The reason for the region server couldn't connect was due to firewall. It
was enabled in my system. And in other machine, It was disabled. I made
sure that firewall is disabled in both the machines and its working perfect
now.

Thank you for the responses.

@pavan, you have to restart your hbase after modifying /etc/hosts.
And, make sure that your /etc/hosts file contains lines ip address and
hostname mapping rows alone comment everything else.
Example:
<ip_address>  <hostname>
<ip_address>  localhost
.
.
And, a famous thing to be noted is replace the 127.0.1.1 with 127.0.0.1
Hope this should solve your problem.


On Thu, Aug 22, 2013 at 8:06 PM, Pavan Sudheendra <pavan0591@gmail.com>wrote:

> And just to be clear, sorry if this is a dumb question.. after updating the
> /etc/hosts file are we supposed to restart hbase?
>
>
> On Thu, Aug 22, 2013 at 8:03 PM, Pavan Sudheendra <pavan0591@gmail.com
> >wrote:
>
> > Isn't hbase.zookeeper.quorum suppose to contain only the address of the
> > HBase master instead of all the region servers?
> >
> >
> >
> > On Thu, Aug 22, 2013 at 8:01 PM, Pavan Sudheendra <pavan0591@gmail.com
> >wrote:
> >
> >> Vamshi and Jay .. Can you both share your /etc/hosts file?
> >>
> >> I have the exact same problem .. All my namenode cluster just log this
> >> connection refused when they are to log something useful for
> de-bugging..
> >> But for me HBase region server tries to connect to localhost when i
> want it
> >> to connect it to its master..
> >>
> >>
> >> On Thu, Aug 22, 2013 at 7:24 PM, Jay Vyas <jayunit100@gmail.com> wrote:
> >>
> >>> Yes this sounds like a zookeeper DNS error.
> >>>
> >>> I just ran into these type of issues a few months ago and wrote up my
> >>> solutions to the 3 main hbase communication/setup errors I got.
> >>>
> >>> See if this helps
> >>>
> http://jayunit100.blogspot.com/2013/05/debugging-hbase-installation.html
> >>>
> >>> Also Make sure iptables are off etc..
> >>>
> >>> On Aug 22, 2013, at 6:02 AM, Vamshi Krishna <vamshi2105@gmail.com>
> >>> wrote:
> >>>
> >>> > Hi I setup a hbase cluster of 2 machines.
> >>> >
> >>> > Master Machine (vamshi_RS) running both master & Regionserver
> >>> > slave machine  - Running only Region server.
> >>> >
> >>> > After i ran start-hbase.sh all the daemons are starting perfectly but
> >>> after
> >>> > some time Regionserver on slave machine  is stopping.
> >>> >
> >>> > I analysed the region server log and  below is the log content.
> >>> > Some how the Region server machine is not able to communicate with
> the
> >>> > zookeeper (I guess). Is that the reason..?
> >>> >
> >>> > Please look at my hbase-site.xml below (after log content), which is
> >>> same
> >>> > in both the machines and kindly let me know the solution for this
> >>> issue.
> >>> >
> >>> >
> >>> > 2013-08-22 14:03:25,023 INFO org.apache.zookeeper.ZooKeeper:
> Initiating
> >>> > client connection, connectString=vamshi_RS:2181 sessionTimeout=180000
> >>> > watcher=regionserver:60020
> >>> > 2013-08-22 14:03:25,033 INFO
> >>> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The
> identifier
> >>> of
> >>> > this process is 7426@vamshi
> >>> > 2013-08-22 14:03:25,038 INFO org.apache.zookeeper.ClientCnxn: Opening
> >>> > socket connection to server vamshi_RS/192.168.1.57:2181. Will not
> >>> attempt
> >>> > to authenticate using SASL (Unable to locate a login configuration)
> >>> > 2013-08-22 14:04:28,171 WARN org.apache.zookeeper.ClientCnxn: Session
> >>> 0x0
> >>> > for server null, unexpected error, closing socket connection and
> >>> attempting
> >>> > reconnect
> >>> > java.net.ConnectException: Connection timed out
> >>> >    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>> >    at
> >>> >
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> >>> >    at
> >>> >
> >>>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> >>> >    at
> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> >>> > 2013-08-22 14:04:28,287 WARN
> >>> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> >>> transient
> >>> > ZooKeeper exception:
> >>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>> > KeeperErrorCode = ConnectionLoss for /hbase/master
> >>> > 2013-08-22 14:04:28,287 INFO
> org.apache.hadoop.hbase.util.RetryCounter:
> >>> > Sleeping 2000ms before retry #1...
> >>> > 2013-08-22 14:04:29,282 INFO org.apache.zookeeper.ClientCnxn: Opening
> >>> > socket connection to server vamshi_RS/192.168.1.57:2181. Will not
> >>> attempt
> >>> > to authenticate using SASL (Unable to locate a login configuration)
> >>> > 2013-08-22 14:05:32,425 WARN org.apache.zookeeper.ClientCnxn: Session
> >>> 0x0
> >>> > for server null, unexpected error, closing socket connection and
> >>> attempting
> >>> > reconnect
> >>> > java.net.ConnectException: Connection timed out
> >>> >    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>> >    at
> >>> >
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> >>> >    at
> >>> >
> >>>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> >>> >    at
> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> >>> > 2013-08-22 14:05:32,526 WARN
> >>> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> >>> transient
> >>> > ZooKeeper exception:
> >>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>> > KeeperErrorCode = ConnectionLoss for /hbase/master
> >>> > 2013-08-22 14:05:32,526 INFO
> org.apache.hadoop.hbase.util.RetryCounter:
> >>> > Sleeping 4000ms before retry #2...
> >>> > 2013-08-22 14:05:33,526 INFO org.apache.zookeeper.ClientCnxn: Opening
> >>> > socket connection to server vamshi_RS/192.168.1.57:2181. Will not
> >>> attempt
> >>> > to authenticate using SASL (Unable to locate a login configuration)
> >>> > 2013-08-22 14:06:36,617 WARN org.apache.zookeeper.ClientCnxn: Session
> >>> 0x0
> >>> > for server null, unexpected error, closing socket connection and
> >>> attempting
> >>> > reconnect
> >>> > java.net.ConnectException: Connection timed out
> >>> >    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>> >    at
> >>> >
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> >>> >    at
> >>> >
> >>>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> >>> >    at
> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> >>> > .
> >>> > .
> >>> > .
> >>> >
> >>> >
> >>> > hbase-site.xml:
> >>> >
> >>> > <property>
> >>> >        <name>hbase.rootdir</name>
> >>> >
> >>> >
> >>>
> <!--value>hdfs://vamshi:54310/home/biginfolabs/BILSftwrs/hbase-0.94.10/data/</value-->
> >>> >    <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/hbstmp/</value>
> >>> >    </property>
> >>> >
> >>> >    <property>
> >>> >        <name>hbase.cluster.distributed</name>
> >>> >        <value>true</value>
> >>> >    </property>
> >>> >    <property>
> >>> >        <name>hbase.master</name>
> >>> >        <value>vamshi_RS</value>
> >>> >    </property>
> >>> >    <property>
> >>> >        <name>hbase.zookeeper.property.clientPort</name>
> >>> >        <value>2181</value>
> >>> >    </property>
> >>> >
> >>> >   <property>
> >>> >        <name>hbase.hregion.max.filesize</name>
> >>> >        <value>50</value>
> >>> >    </property>
> >>> >
> >>> >   <property>
> >>> >        <name>hbase.balancer.period</name>
> >>> >        <value>60000</value>
> >>> >    </property>
> >>> >
> >>> >    <property>
> >>> >        <name>hbase.zookeeper.quorum</name>
> >>> >        <value>vamshi_RS</value>
> >>> >    </property>
> >>> >    <property>
> >>> >        <name>hbase.zookeeper.property.dataDir</name>
> >>> >
>  <value>/home/biginfolabs/BILSftwrs/hbase-0.94.10/zkptmp</value>
> >>> >    </property>
> >>> >  <property>
> >>> >    <name>hbase.client.scanner.caching</name>
> >>> >    <value>1000</value>
> >>> >    <description>Number of rows that will be fetched when calling next
> >>> >    </description>
> >>> >  </property>
> >>> >  <property>
> >>> >    <name>hbase.zookeeper.property.maxClientCnxns</name>
> >>> >    <value>1024</value>
> >>> >  </property>
> >>> >
> >>> > <property>
> >>> >    <name>hbase.coprocessor.user.region.classes</name>
> >>> >    <value>com.bil.coproc.ColumnAggregationEndpoint</value>
> >>> >  </property>
> >>> >
> >>> > --
> >>> > *Regards*
> >>> > *
> >>> > Vamshi Krishna
> >>> > *
> >>>
> >>
> >>
> >>
> >> --
> >> Regards-
> >> Pavan
> >>
> >
> >
> >
> > --
> > Regards-
> > Pavan
> >
>
>
>
> --
> Regards-
> Pavan
>



-- 
*Regards*
*
Vamshi Krishna
*

--047d7b47215e9df61604e49afdc3--