hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yahoo <mike.dow...@gmail.com>
Subject Re: Disk usage drops after RegionServer restart? (0.98)
Date Wed, 28 Oct 2015 22:20:24 GMT
I see exactly the same thing on one of our clusters, also running HBase 
0.98 (not sure of the rest of the version number since I'm not in the 
office right now). The non-hdfs disk space slowly fills up and I failed 
to locate the actual files using 'du'. I did a lot of googling but 
couldn't find any other mentions of the problem at the time.

Mike.

On 24/10/2015 04:08, Otis Gospodnetić wrote:
> Hi Ted,
>
> 0.98.6-cdh5.3.0
>
> I did actually try to use lsof, but I didn't see anything unusual there.
> Is there something specific I should look for?  Things owned by hbase user
> or hdfs or yarn?  Hm, here, I don't really see anything interesting
<snip>
>
> Thanks,
> Otis
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
> On Fri, Oct 23, 2015 at 10:26 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> Which specific release of 0.98 are you using ?
>>
>> Have you used lsof to see which files were being held onto ?
>>
>> Thanks
>>
>> On Fri, Oct 23, 2015 at 7:21 PM, Otis Gospodnetić <
>> otis.gospodnetic@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> Is/was there a known issue with HBase 0.98 "holding onto" files?
>>>
>>> We noticed the used disk space metric going up, up and up and we could
>> not
>>> stop it with major compaction.
>>> But we noticed that if we restart a RegionServer 2 things happen:
>>> 1) its disk usage immediately drops a lot
>>> 2) the disk usage of other RegionServers drops some as well
>>>
>>> Have a look at this chart:
>>>    https://apps.sematext.com/spm-reports/s/Ssy4ViFGHq
>>>
>>> At 1:54 we restarted the first RS (blue line)
>>> At 2:03 we restarted the second RS (dark green line)
>>>
>>> Is/was this a known HBase 0.98 issue?
>>>
>>> Thanks,
>>> Otis
>>> --
>>> Monitoring - Log Management - Alerting - Anomaly Detection
>>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>>


Mime
View raw message