hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: HBase Region Server Spinning on JMX requests
Date Wed, 24 Apr 2013 16:31:01 GMT
This is probably unprotected concurrent access to a HashMap in Hadoop
metrics. See comments on https://issues.apache.org/jira/browse/HBASE-8416


On Wed, Apr 24, 2013 at 4:37 AM, Buckley,Ron <buckleyr@oclc.org> wrote:

> I created https://issues.apache.org/jira/browse/HBASE-8416
>
> We're not using OpenTSDB, but we do have something similar grabbing the
> jmx data on a regular basis.
>
> Eventually, we moved all the regions off of that region server.  We left
> it spinning overnight, going to try to look at it this morning.
>
>
> -----Original Message-----
> From: Kevin O'dell [mailto:kevin.odell@cloudera.com]
> Sent: Tuesday, April 23, 2013 11:04 PM
> To: user@hbase.apache.org; lars hofhansl
> Subject: Re: HBase Region Server Spinning on JMX requests
>
> Hi Ron,
>
>   Are you using OpenTSDB?  I have seen:
>
> https://issues.apache.org/jira/browse/HBASE-6602 (which should be
> addressed in your build).  One possibility is that the Tcollector is
> leaving lots of connections open and causing the spin.  Unfortunately,
> we have not been able to nail it down further.  We are thinking
> Metrics2 in trunk might inadvertently take care of this issue.
>
> On Tue, Apr 23, 2013 at 6:57 PM, lars hofhansl <larsh@apache.org> wrote:
> > Hmm... That's not good. Would you mind filing a ticket here:
> https://issues.apache.org/jira/browse/HBASE ?
> >
> > -- Lars
> >
> >
> > ________________________________
> >  From: "Buckley,Ron" <buckleyr@oclc.org>
> > To: user@hbase.apache.org
> > Sent: Tuesday, April 23, 2013 6:57 AM
> > Subject: HBase Region Server Spinning on JMX requests
> >
> >
> > This is with HBase 0.94.4 & CDH 4.1.1
> >
> > This morning one our region servers (we have 44) stopped responding to
> > the '/jmx' request. (It's working for regular activity.)
> Additionally,
> > the region server is now using all the CPU on the host, running all 8
> > cores at 100%.
> >
> > I've got several jstacks, they all look like this:
> > http://pastebin.com/dGTmTEN7
> >
> > If I do a wget of the /jmx url, it starts responding, but never
> > completes, always stopping at the same point:
> > http://pastebin.com/qhNvxrQK
> >
> > Has anyone ever seen this before? If so, Is there a way out of it?
> > (other than bouncing the region server).
> >
> > BTW: There's nothing relevant in the region server log and the garbage
> > collector log is normal.
> >
> >
> > ----------------------------------------------------------------------
> > Ron Buckley
>
>
>
> --
> Kevin O'Dell
> Systems Engineer, Cloudera
>
>
>


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message