hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Esteban Gutierrez <este...@cloudera.com>
Subject Re: RegionServer many socket fds are in CLOSE_WAIT and not getting cleared
Date Thu, 24 Jul 2014 18:00:27 GMT
Shankar,

seems that those close_wait are bound to ipv6, have you tried to disable
ipv6 first? (-Djava.net.preferIPv4Stack=true)

Regards,
Esteban.



--
Cloudera, Inc.



On Thu, Jul 24, 2014 at 5:02 AM, Shankar hiremath <
shankar.hiremath@huawei.com> wrote:

> I am using Hbase 0.98.3 and Hadoop 2.4.1
>
> Regards
> -Shankar
>
>
> This e-mail and its attachments contain confidential information from
> HUAWEI, which is intended only for the person or entity whose address is
> listed above. Any use of the information contained herein in any way
> (including, but not limited to, total or partial disclosure, reproduction,
> or dissemination) by persons other than the intended recipient(s) is
> prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!
>
>
>
>
>
> -----Original Message-----
> From: Qiang Tian [mailto:tianq01@gmail.com]
> Sent: 24 July 2014 PM 03:19
> To: user@hbase.apache.org
> Subject: Re: RegionServer many socket fds are in CLOSE_WAIT and not
> getting cleared
>
> what is your hbase version?
> the close_wait is in regionserver process, probably rpc connection close()
> is not call due to some reason? can you pastebin the related regionserver
> log?
>
>
> On Thu, Jul 24, 2014 at 3:16 PM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > Sorry about that.
> > https://issues.apache.org/jira/browse/HBASE-9393.  Similar issue has
> > been reported.
> >
> > What version of HBase and HDFS are you using?  See if things are similar.
> >
> > Regards
> > Ram
> >
> >
> > On Thu, Jul 24, 2014 at 12:33 PM, Shankar hiremath <
> > shankar.hiremath@huawei.com> wrote:
> >
> > > Hi Ram,
> > >
> > > Actually It is not file FD, it is socket FD which is in CLOSE_WAIT
> > > state (tcp socket connection)
> > >
> > > > java    18028 shankar1  194u  IPv6 18548560      0t0  TCP
> > > > XX-XX-XX-XX:62675->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > >
> > > Regards
> > > -Shankar
> > >
> > >
> > > This e-mail and its attachments contain confidential information
> > > from HUAWEI, which is intended only for the person or entity whose
> > > address is listed above. Any use of the information contained herein
> > > in any way (including, but not limited to, total or partial
> > > disclosure,
> > reproduction,
> > > or dissemination) by persons other than the intended recipient(s) is
> > > prohibited. If you receive this e-mail in error, please notify the
> > > sender by phone or email immediately and delete it!
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: ramkrishna vasudevan [mailto:ramkrishna.s.vasudevan@gmail.com]
> > > Sent: 24 July 2014 PM 12:23
> > > To: user@hbase.apache.org
> > > Subject: Re: RegionServer many socket fds are in CLOSE_WAIT and not
> > > getting cleared
> > >
> > > Does your regions have store files in them?
> > > In HBase all the store files are opened and the readers are kept
> > > open always when the region opening happens.  When there is a
> > > compaction happening the count of those file handlers open gets
> > > reduced based on the number of files compacted.
> > >
> > > Regards
> > > Ram
> > >
> > >
> > > On Thu, Jul 24, 2014 at 12:19 PM, Shankar hiremath <
> > > shankar.hiremath@huawei.com> wrote:
> > >
> > > > Dear All,
> > > >
> > > > Observation:
> > > > I have HBase cluster with Kerberos enabled, when the Region Server
> > > > startups then we observed some 20-30 socket fds are in CLOSE_WAIT
> > > > state, After that when the Region Server starts opening then the
> > > > socket fds in CLOSE_WAIT starts increasing gradually (almost same
> > > > as number of regions opened by the region server) And all these
> > > > CLOSE_WAIT states are not getting cleared up,
> > > >
> > > > /hbase> lsof -i | grep `jps | grep RegionServer | cut -d " " -f1`
> > > > java    18028 shankar1  118u  IPv6 18552894      0t0  TCP
> > > > XX-XX-XX-XX:60020 (LISTEN)
> > > > java    18028 shankar1  160u  IPv6 18548520      0t0  TCP *:60030
> > > (LISTEN)
> > > > java    18028 shankar1  167u  IPv6 18548522      0t0  TCP
> > > > XX-XX-XX-XX:42534-> XX-XX-XX-XX:eforward (ESTABLISHED)
> > > > java    18028 shankar1  172u  IPv6 18552916      0t0  TCP
> > > > XX-XX-XX-XX:42535-> XX-XX-XX-XX:eforward (ESTABLISHED)
> > > > java    18028 shankar1  173u  IPv6 18551227      0t0  TCP
> > > > XX-XX-XX-XX:49646->XX-XX-XX-XX:60000 (ESTABLISHED)
> > > > java    18028 shankar1  178u  IPv6 18551237      0t0  TCP
> > > > XX-XX-XX-XX:62668->XX-XX-XX-XX:busboy (ESTABLISHED)
> > > > java    18028 shankar1  185u  IPv6 18548549      0t0  TCP
> > > > XX-XX-XX-XX:21856-> XX-XX-XX-XX:eforward (ESTABLISHED)
> > > > java    18028 shankar1  187u  IPv6 18548558      0t0  TCP
> > > > XX-XX-XX-XX:62673->XX-XX-XX-XX:busboy (ESTABLISHED)
> > > > java    18028 shankar1  188u  IPv6 18601323      0t0  TCP
> > > > XX-XX-XX-XX:63168->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > > java    18028 shankar1  189u  IPv6 18601322      0t0  TCP
> > > > XX-XX-XX-XX:63167->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > > java    18028 shankar1  190u  IPv6 18601324      0t0  TCP
> > > > XX-XX-XX-XX:63169->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > > java    18028 shankar1  191r  IPv6 18592423      0t0  TCP
> > > > XX-XX-XX-XX:63087->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > > java    18028 shankar1  193u  IPv6 18593210      0t0  TCP
> > > > XX-XX-XX-XX:63090->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > > java    18028 shankar1  194u  IPv6 18548560      0t0  TCP
> > > > XX-XX-XX-XX:62675->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > > java    18028 shankar1  195u  IPv6 18592428      0t0  TCP
> > > > XX-XX-XX-XX:63093->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > > java    18028 shankar1  196u  IPv6 18593218      0t0  TCP
> > > > XX-XX-XX-XX:63096->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > > java    18028 shankar1  197u  IPv6 18591423      0t0  TCP
> > > > XX-XX-XX-XX:63105->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > > java    18028 shankar1  201u  IPv6 18592431      0t0  TCP
> > > > XX-XX-XX-XX:63099->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > > java    18028 shankar1  202u  IPv6 18592433      0t0  TCP
> > > > XX-XX-XX-XX:63102->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > > java    18028 shankar1  203u  IPv6 18552317      0t0  TCP
> > > > XX-XX-XX-XX:62681->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > >
> > > >
> > >
> >
> ............................................................................................................
> > > >
> > > >
> > >
> >
> ............................................................................................................
> > > >
> > > >
> > >
> >
> ............................................................................................................
> > > >
> > > >
> > >
> >
> ............................................................................................................
> > > >
> > > >
> > >
> >
> ............................................................................................................
> > > >
> > > > any input or suggestion will be helpful, or is it a bug.
> > > >
> > > > Regards
> > > > -Shankar
> > > >
> > > >
> > > > [X]
> > > > This e-mail and its attachments contain confidential information
> > > > from HUAWEI, which is intended only for the person or entity whose
> > > > address is listed above. Any use of the information contained
> > > > herein in any way (including, but not limited to, total or partial
> > > > disclosure, reproduction, or dissemination) by persons other than
> > > > the intended
> > > > recipient(s) is prohibited. If you receive this e-mail in error,
> > > > please notify the sender by phone or email immediately and delete it!
> > > > [X]
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message