Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of tianq01@gmail.com designates
 209.85.192.51 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAT7Mko1WCxYh2bm1xFP-mf6rYeG+ejoQWz=ccZnsMRkbu+uRQ@mail.gmail.com>
References: 
 <CB0197544979BE458C310023C8F4505B091B81BA@nkgeml510-mbs.china.huawei.com>
	<CAAT7MkpuW+o+Q_1_br6-2O7to9WhJPvt6SP5XE=OHkF0sCu3fw@mail.gmail.com>
	<CB0197544979BE458C310023C8F4505B091B8365@nkgeml510-mbs.china.huawei.com>
	<CAAT7Mko1WCxYh2bm1xFP-mf6rYeG+ejoQWz=ccZnsMRkbu+uRQ@mail.gmail.com>
Date: Thu, 24 Jul 2014 17:48:31 +0800
Message-ID: 
 <CAMZUsP4Dnxu1vsyp6yC6+Do=ouv1xPwwmkOdikw1KxmqC-Gj0w@mail.gmail.com>
Subject: Re: RegionServer many socket fds are in CLOSE_WAIT and not getting
 cleared
From: Qiang Tian <tianq01@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a113987200513f404feed5cd7

--001a113987200513f404feed5cd7
Content-Type: text/plain; charset=UTF-8

what is your hbase version?
the close_wait is in regionserver process, probably rpc connection close()
is not call due to some reason? can you pastebin the related regionserver
log?


On Thu, Jul 24, 2014 at 3:16 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:

> Sorry about that.
> https://issues.apache.org/jira/browse/HBASE-9393.  Similar issue has been
> reported.
>
> What version of HBase and HDFS are you using?  See if things are similar.
>
> Regards
> Ram
>
>
> On Thu, Jul 24, 2014 at 12:33 PM, Shankar hiremath <
> shankar.hiremath@huawei.com> wrote:
>
> > Hi Ram,
> >
> > Actually It is not file FD, it is socket FD which is in CLOSE_WAIT state
> > (tcp socket connection)
> >
> > > java    18028 shankar1  194u  IPv6 18548560      0t0  TCP
> > > XX-XX-XX-XX:62675->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> >
> > Regards
> > -Shankar
> >
> >
> > This e-mail and its attachments contain confidential information from
> > HUAWEI, which is intended only for the person or entity whose address is
> > listed above. Any use of the information contained herein in any way
> > (including, but not limited to, total or partial disclosure,
> reproduction,
> > or dissemination) by persons other than the intended recipient(s) is
> > prohibited. If you receive this e-mail in error, please notify the sender
> > by phone or email immediately and delete it!
> >
> >
> >
> >
> > -----Original Message-----
> > From: ramkrishna vasudevan [mailto:ramkrishna.s.vasudevan@gmail.com]
> > Sent: 24 July 2014 PM 12:23
> > To: user@hbase.apache.org
> > Subject: Re: RegionServer many socket fds are in CLOSE_WAIT and not
> > getting cleared
> >
> > Does your regions have store files in them?
> > In HBase all the store files are opened and the readers are kept open
> > always when the region opening happens.  When there is a compaction
> > happening the count of those file handlers open gets reduced based on the
> > number of files compacted.
> >
> > Regards
> > Ram
> >
> >
> > On Thu, Jul 24, 2014 at 12:19 PM, Shankar hiremath <
> > shankar.hiremath@huawei.com> wrote:
> >
> > > Dear All,
> > >
> > > Observation:
> > > I have HBase cluster with Kerberos enabled, when the Region Server
> > > startups then we observed some 20-30 socket fds are in CLOSE_WAIT
> > > state, After that when the Region Server starts opening then the
> > > socket fds in CLOSE_WAIT starts increasing gradually (almost same as
> > > number of regions opened by the region server) And all these
> > > CLOSE_WAIT states are not getting cleared up,
> > >
> > > /hbase> lsof -i | grep `jps | grep RegionServer | cut -d " " -f1`
> > > java    18028 shankar1  118u  IPv6 18552894      0t0  TCP
> > > XX-XX-XX-XX:60020 (LISTEN)
> > > java    18028 shankar1  160u  IPv6 18548520      0t0  TCP *:60030
> > (LISTEN)
> > > java    18028 shankar1  167u  IPv6 18548522      0t0  TCP
> > > XX-XX-XX-XX:42534-> XX-XX-XX-XX:eforward (ESTABLISHED)
> > > java    18028 shankar1  172u  IPv6 18552916      0t0  TCP
> > > XX-XX-XX-XX:42535-> XX-XX-XX-XX:eforward (ESTABLISHED)
> > > java    18028 shankar1  173u  IPv6 18551227      0t0  TCP
> > > XX-XX-XX-XX:49646->XX-XX-XX-XX:60000 (ESTABLISHED)
> > > java    18028 shankar1  178u  IPv6 18551237      0t0  TCP
> > > XX-XX-XX-XX:62668->XX-XX-XX-XX:busboy (ESTABLISHED)
> > > java    18028 shankar1  185u  IPv6 18548549      0t0  TCP
> > > XX-XX-XX-XX:21856-> XX-XX-XX-XX:eforward (ESTABLISHED)
> > > java    18028 shankar1  187u  IPv6 18548558      0t0  TCP
> > > XX-XX-XX-XX:62673->XX-XX-XX-XX:busboy (ESTABLISHED)
> > > java    18028 shankar1  188u  IPv6 18601323      0t0  TCP
> > > XX-XX-XX-XX:63168->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java    18028 shankar1  189u  IPv6 18601322      0t0  TCP
> > > XX-XX-XX-XX:63167->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java    18028 shankar1  190u  IPv6 18601324      0t0  TCP
> > > XX-XX-XX-XX:63169->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java    18028 shankar1  191r  IPv6 18592423      0t0  TCP
> > > XX-XX-XX-XX:63087->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java    18028 shankar1  193u  IPv6 18593210      0t0  TCP
> > > XX-XX-XX-XX:63090->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java    18028 shankar1  194u  IPv6 18548560      0t0  TCP
> > > XX-XX-XX-XX:62675->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java    18028 shankar1  195u  IPv6 18592428      0t0  TCP
> > > XX-XX-XX-XX:63093->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java    18028 shankar1  196u  IPv6 18593218      0t0  TCP
> > > XX-XX-XX-XX:63096->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java    18028 shankar1  197u  IPv6 18591423      0t0  TCP
> > > XX-XX-XX-XX:63105->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java    18028 shankar1  201u  IPv6 18592431      0t0  TCP
> > > XX-XX-XX-XX:63099->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java    18028 shankar1  202u  IPv6 18592433      0t0  TCP
> > > XX-XX-XX-XX:63102->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java    18028 shankar1  203u  IPv6 18552317      0t0  TCP
> > > XX-XX-XX-XX:62681->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > >
> > >
> >
> ............................................................................................................
> > >
> > >
> >
> ............................................................................................................
> > >
> > >
> >
> ............................................................................................................
> > >
> > >
> >
> ............................................................................................................
> > >
> > >
> >
> ............................................................................................................
> > >
> > > any input or suggestion will be helpful, or is it a bug.
> > >
> > > Regards
> > > -Shankar
> > >
> > >
> > > [X]
> > > This e-mail and its attachments contain confidential information from
> > > HUAWEI, which is intended only for the person or entity whose address
> > > is listed above. Any use of the information contained herein in any
> > > way (including, but not limited to, total or partial disclosure,
> > > reproduction, or dissemination) by persons other than the intended
> > > recipient(s) is prohibited. If you receive this e-mail in error,
> > > please notify the sender by phone or email immediately and delete it!
> > > [X]
> > >
> > >
> > >
> > >
> > >
> > >
> >
>

--001a113987200513f404feed5cd7--