hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lars hofhansl <la...@apache.org>
Subject Re: master consumes large amount of CPU for days
Date Thu, 26 Mar 2015 22:43:03 GMT
Hi Ted,

Yes, it is safe to bounce the HMaster without taking the region servers down.

Are there any regions in transition (would be shown on the master's page)?

All of the threads involved with
- org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(org.apache.hadoop.fs.Path) @bci=14,
line=309 (Compiled frame)
  ...
- org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(java.util.List) @bci=50, line=1932
(Compiled frame)

are a bit suspect. Who's requesting all the table descriptors from the master?

-- Lars
     From: Ted Tuttle <ted@mentacapital.com>
 To: "user@hbase.apache.org" <user@hbase.apache.org> 
Cc: Development <Development@mentacapital.com> 
 Sent: Thursday, March 26, 2015 12:18 PM
 Subject: master consumes large amount of CPU for days
   
Hello-

Our master process started consuming a large amount of CPU (75% of box) several days back
and hasn't stopped.  I have 2 questions:

                1) what is it doing (stack dump and log below)
                2) is it safe to restart the master without taking the whole cluster
down?

Master stack dump:

                http://pastebin.com/G0iNNEpC

Master log from last 15 mins or so:

                http://pastebin.com/WQNjhFGf

Thanks,
Ted


  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message