Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of zhongresearch@gmail.com
 designates 209.85.223.188 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=dy7H0fa24BSUko3m1Nubl5BsFlFwurxbJGV2MqM2w1Y7Vf9kEXnVjna7HRsJ9yzEwW
         9lfh6F35Se/AVItOMKW4VHP9VuDLkwwF/zq7IFu0H4TF7MtI08yd631fITcSK6aq81b1
         pK2/vnvT9APwiO0VSDbLoEZuXMogVfwygC8As=
MIME-Version: 1.0
In-Reply-To: <7c962aed0911141203l100e5e43r284b18b9958f5838@mail.gmail.com>
References: <dea827c60911110902y4fd83f03mc0a3e216085697ad@mail.gmail.com>
	 <dea827c60911131326y2d2d7bbepd0b651fd7b9815fd@mail.gmail.com>
	 <80942B37-2300-489C-A8A5-AFD3F267CD9E@gmail.com>
	 <dea827c60911131337q78d7c39cuc225e344cc134351@mail.gmail.com>
	 <7c962aed0911131502g5960797q10d8cdff89ecbe85@mail.gmail.com>
	 <dea827c60911131544o9351b39ra81ab12480ebe35@mail.gmail.com>
	 <7c962aed0911131608x52317c12j769305788cc6e1b6@mail.gmail.com>
	 <dea827c60911132201l1eb6655alb34b27169dcbd573@mail.gmail.com>
	 <dea827c60911140831h64edfe1ahba1736c80b55724d@mail.gmail.com>
	 <7c962aed0911141203l100e5e43r284b18b9958f5838@mail.gmail.com>
Date: Sat, 14 Nov 2009 16:20:15 -0500
Message-ID: <dea827c60911141320s150a3526i4b53e4ec1d383c00@mail.gmail.com>
Subject: Re: regionserver disconnection
From: Zhenyu Zhong <zhongresearch@gmail.com>
To: hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64603eab0d47004785b5735

--0016e64603eab0d47004785b5735
Content-Type: text/plain; charset=ISO-8859-1

Stack,

I am very appreciated for your help.
I went back to check the GC log.
I found one entry that is
8677.032: [CMS-concurrent-mark: 1.547/2170.359 secs] [Times: user=50.29
sys=3.65, real=2170.49 secs]
I couldn't tell what time this entry was logged.

This is the only entry I found that has such huge number of GC time.
I also saw a few long GC time around 200 secs. (should be within the session
timeout)

Next, I need to find a way to correlate the timestamp with a friendly
dateformat on each GC log entry.

Hopefully this is the reason.

Best,
zhenyu


On Sat, Nov 14, 2009 at 3:03 PM, stack <stack@duboce.net> wrote:

> On Sat, Nov 14, 2009 at 8:31 AM, Zhenyu Zhong <zhongresearch@gmail.com
> >wrote:
>
> ....
>
> > 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > response for sessionid:0x424f0ed01ea00a6 after 0ms
> > 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn: Got ping
> > response for sessionid:0x424f0ed01ea00a6 after 2010ms
> > 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn: Exception
> > closing session 0x424f0ed01ea00a6 to sun.nio.ch.SelectionKeyImpl@39fbb2d6
> > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> > lim=4 cap=4]
> >        at
> > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> >
>
> Anything happening on the machine at this time?  The ping time is starting
> to climb... was 0ms for a long time and the last log is > 2seconds.  Whats
> in the GC logs?  Anything else running on this machine?
> St.Ack
>

--0016e64603eab0d47004785b5735--