hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Disk usage drops after RegionServer restart? (0.98)
Date Thu, 29 Oct 2015 08:48:41 GMT
Interesting 

By same config I guess you mean same hbase config. 
Can you find out what was different between the two clusters ?

Thanks

> On Oct 29, 2015, at 1:26 AM, Loïc Chanel <loic.chanel@telecomnancy.net> wrote:
> 
> I can see that too on one of our clusters, and the thing which is really
> weird is that another one of ours has the exact same configuration (as it
> is the pre-production cluster) and we don't see the problem there.
> I also did a lot of googling, but as we couldn't find a solution we simply
> made a cron to restart periodically the RegionServers (to avoid a full on
> Hadoop data partitions).
> 
> Regards,
> 
> 
> Loïc
> 
> Loïc CHANEL
> System & virtualization engineer
> TO - XaaS Ind - Worldline (Villeurbanne, France)
> 
> 2015-10-28 23:20 GMT+01:00 Yahoo <mike.downey@gmail.com>:
> 
>> I see exactly the same thing on one of our clusters, also running HBase
>> 0.98 (not sure of the rest of the version number since I'm not in the
>> office right now). The non-hdfs disk space slowly fills up and I failed to
>> locate the actual files using 'du'. I did a lot of googling but couldn't
>> find any other mentions of the problem at the time.
>> 
>> Mike.
>> 
>>> On 24/10/2015 04:08, Otis Gospodnetić wrote:
>>> 
>>> Hi Ted,
>>> 
>>> 0.98.6-cdh5.3.0
>>> 
>>> I did actually try to use lsof, but I didn't see anything unusual there.
>>> Is there something specific I should look for?  Things owned by hbase user
>>> or hdfs or yarn?  Hm, here, I don't really see anything interesting
>> <snip>
>> 
>> 
>>> Thanks,
>>> Otis
>>> --
>>> Monitoring - Log Management - Alerting - Anomaly Detection
>>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>> 
>>> 
>>> On Fri, Oct 23, 2015 at 10:26 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>> 
>>> Which specific release of 0.98 are you using ?
>>>> 
>>>> Have you used lsof to see which files were being held onto ?
>>>> 
>>>> Thanks
>>>> 
>>>> On Fri, Oct 23, 2015 at 7:21 PM, Otis Gospodnetić <
>>>> otis.gospodnetic@gmail.com> wrote:
>>>> 
>>>> Hello,
>>>>> 
>>>>> Is/was there a known issue with HBase 0.98 "holding onto" files?
>>>>> 
>>>>> We noticed the used disk space metric going up, up and up and we could
>>>> not
>>>> 
>>>>> stop it with major compaction.
>>>>> But we noticed that if we restart a RegionServer 2 things happen:
>>>>> 1) its disk usage immediately drops a lot
>>>>> 2) the disk usage of other RegionServers drops some as well
>>>>> 
>>>>> Have a look at this chart:
>>>>>   https://apps.sematext.com/spm-reports/s/Ssy4ViFGHq
>>>>> 
>>>>> At 1:54 we restarted the first RS (blue line)
>>>>> At 2:03 we restarted the second RS (dark green line)
>>>>> 
>>>>> Is/was this a known HBase 0.98 issue?
>>>>> 
>>>>> Thanks,
>>>>> Otis
>>>>> --
>>>>> Monitoring - Log Management - Alerting - Anomaly Detection
>>>>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>> 

Mime
View raw message