hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Re: taking snapshot's creates to many TCP CLOSE_WAIT handles on the hbase master server
Date Tue, 22 Apr 2014 16:21:44 GMT
I went over the jstack output.
There were 53 IPC Server handler threads which were mostly in WAITING state.

Please try Stack's suggestion and see if the problem gets resolved.

Cheers


On Tue, Apr 22, 2014 at 3:18 AM, Hansi Klose <hansi.klose@web.de> wrote:

> Hi Ted,
>
> I inserted the output at pastebin
>
> http://pastebin.com/n3mMPxBA
>
> At the moment the hbase master process holds 10716 handles.
> We stopped making snapshots last week.
> After 4 days the count is still the same.
>
> Regards Hansi
>
> > Gesendet: Donnerstag, 17. April 2014 um 19:09 Uhr
> > Von: "Ted Yu" <yuzhihong@gmail.com>
> > An: "user@hbase.apache.org" <user@hbase.apache.org>
> > Betreff: Re: taking snapshot's creates to many TCP CLOSE_WAIT handles on
> the hbase master server
> >
> > Can you take jstack of master process and pastebin it ?
> >
> > Thanks
> >
> >
> > On Thu, Apr 17, 2014 at 6:51 AM, Hansi Klose <hansi.klose@web.de> wrote:
> >
> > > Hi,
> > >
> > > we use a script to take on a regular basis snapshot's and delete old
> one's.
> > >
> > > We recognizes that the web interface of the hbase master was not
> working
> > > any more becaues of too many open files.
> > >
> > > The master reaches his number of open file limit of 32768
> > >
> > > When I run lsof I saw that there where a lot of TCP CLOSE_WAIT handles
> open
> > > with the regionserver as target.
> > >
> > > On the regionserver there is just one connection to the hbase master.
> > >
> > > I can see that the count of the CLOSE_WAIT handles grow each time
> > > i take a snapshot. When i delete on nothing changes.
> > > Each time i take a snapshot  there are 20 - 30 new CLOSE_WAIT handles.
> > >
> > > Why does the master do not close the handles? Is there a parameter
> > > with a timeout we can use?
> > >
> > > We use hbase 0.94.2-cdh4.2.0.
> > >
> > > Regards Hansi
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message