hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Loïc Chanel <loic.cha...@telecomnancy.net>
Subject Re: Disk usage drops after RegionServer restart? (0.98)
Date Thu, 29 Oct 2015 08:50:39 GMT
Here is the problem : the config is the exact same for HBase and anything
Hadoop or system-related into the cluster.
There is no apparent difference.

Loïc

Loïc CHANEL
System & virtualization engineer
TO - XaaS Ind - Worldline (Villeurbanne, France)

2015-10-29 9:48 GMT+01:00 Ted Yu <yuzhihong@gmail.com>:

> Interesting
>
> By same config I guess you mean same hbase config.
> Can you find out what was different between the two clusters ?
>
> Thanks
>
> > On Oct 29, 2015, at 1:26 AM, Loïc Chanel <loic.chanel@telecomnancy.net>
> wrote:
> >
> > I can see that too on one of our clusters, and the thing which is really
> > weird is that another one of ours has the exact same configuration (as it
> > is the pre-production cluster) and we don't see the problem there.
> > I also did a lot of googling, but as we couldn't find a solution we
> simply
> > made a cron to restart periodically the RegionServers (to avoid a full on
> > Hadoop data partitions).
> >
> > Regards,
> >
> >
> > Loïc
> >
> > Loïc CHANEL
> > System & virtualization engineer
> > TO - XaaS Ind - Worldline (Villeurbanne, France)
> >
> > 2015-10-28 23:20 GMT+01:00 Yahoo <mike.downey@gmail.com>:
> >
> >> I see exactly the same thing on one of our clusters, also running HBase
> >> 0.98 (not sure of the rest of the version number since I'm not in the
> >> office right now). The non-hdfs disk space slowly fills up and I failed
> to
> >> locate the actual files using 'du'. I did a lot of googling but couldn't
> >> find any other mentions of the problem at the time.
> >>
> >> Mike.
> >>
> >>> On 24/10/2015 04:08, Otis Gospodnetić wrote:
> >>>
> >>> Hi Ted,
> >>>
> >>> 0.98.6-cdh5.3.0
> >>>
> >>> I did actually try to use lsof, but I didn't see anything unusual
> there.
> >>> Is there something specific I should look for?  Things owned by hbase
> user
> >>> or hdfs or yarn?  Hm, here, I don't really see anything interesting
> >> <snip>
> >>
> >>
> >>> Thanks,
> >>> Otis
> >>> --
> >>> Monitoring - Log Management - Alerting - Anomaly Detection
> >>> Solr & Elasticsearch Consulting Support Training -
> http://sematext.com/
> >>>
> >>>
> >>> On Fri, Oct 23, 2015 at 10:26 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>>
> >>> Which specific release of 0.98 are you using ?
> >>>>
> >>>> Have you used lsof to see which files were being held onto ?
> >>>>
> >>>> Thanks
> >>>>
> >>>> On Fri, Oct 23, 2015 at 7:21 PM, Otis Gospodnetić <
> >>>> otis.gospodnetic@gmail.com> wrote:
> >>>>
> >>>> Hello,
> >>>>>
> >>>>> Is/was there a known issue with HBase 0.98 "holding onto" files?
> >>>>>
> >>>>> We noticed the used disk space metric going up, up and up and we
> could
> >>>> not
> >>>>
> >>>>> stop it with major compaction.
> >>>>> But we noticed that if we restart a RegionServer 2 things happen:
> >>>>> 1) its disk usage immediately drops a lot
> >>>>> 2) the disk usage of other RegionServers drops some as well
> >>>>>
> >>>>> Have a look at this chart:
> >>>>>   https://apps.sematext.com/spm-reports/s/Ssy4ViFGHq
> >>>>>
> >>>>> At 1:54 we restarted the first RS (blue line)
> >>>>> At 2:03 we restarted the second RS (dark green line)
> >>>>>
> >>>>> Is/was this a known HBase 0.98 issue?
> >>>>>
> >>>>> Thanks,
> >>>>> Otis
> >>>>> --
> >>>>> Monitoring - Log Management - Alerting - Anomaly Detection
> >>>>> Solr & Elasticsearch Consulting Support Training -
> http://sematext.com/
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message