hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Esteban Gutierrez <este...@cloudera.com>
Subject Re: master consumes large amount of CPU for days
Date Sat, 28 Mar 2015 06:21:10 GMT
Hi Ted,

Since HBase 0.98.8 we do better caching those calls (see HBASE-12219) can
you upgrade to that version? Or if you are using 0.94 and you can't upgrade
to HBase 1 or 0.98 there is a patch in that JIRA for 0.94 that might
improve the situation.

cheers,
esteban.


--
Cloudera, Inc.


On Thu, Mar 26, 2015 at 4:22 PM, Ted Tuttle <ted@mentacapital.com> wrote:

> We have no regions in transition.
>
> As far as who's requesting the table descriptors:  We are calling
> HTableIterface.getTableDescriptor frequently.  We are going to cache the
> result to avoid the unnecessary network calls.
>
> Thanks for the hint!
>
> From: lars hofhansl [mailto:larsh@apache.org]
> Sent: Thursday, March 26, 2015 3:43 PM
> To: user@hbase.apache.org
> Cc: Development
> Subject: Re: master consumes large amount of CPU for days
>
> Hi Ted,
>
> Yes, it is safe to bounce the HMaster without taking the region servers
> down.
>
> Are there any regions in transition (would be shown on the master's page)?
>
> All of the threads involved with
> -
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(org.apache.hadoop.fs.Path)
> @bci=14, line=309 (Compiled frame)
>   ...
> -
> org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(java.util.List)
> @bci=50, line=1932 (Compiled frame)
>
> are a bit suspect. Who's requesting all the table descriptors from the
> master?
> -- Lars
>
> ________________________________
> From: Ted Tuttle <ted@mentacapital.com<mailto:ted@mentacapital.com>>
> To: "user@hbase.apache.org<mailto:user@hbase.apache.org>" <
> user@hbase.apache.org<mailto:user@hbase.apache.org>>
> Cc: Development <Development@mentacapital.com<mailto:
> Development@mentacapital.com>>
> Sent: Thursday, March 26, 2015 12:18 PM
> Subject: master consumes large amount of CPU for days
>
> Hello-
>
> Our master process started consuming a large amount of CPU (75% of box)
> several days back and hasn't stopped.  I have 2 questions:
>
>                 1) what is it doing (stack dump and log below)
>                 2) is it safe to restart the master without taking the
> whole cluster down?
>
> Master stack dump:
>
>                 http://pastebin.com/G0iNNEpC
>
> Master log from last 15 mins or so:
>
>                 http://pastebin.com/WQNjhFGf
>
> Thanks,
> Ted
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message