hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Re: oldWALs: what it is and how can I clean it?
Date Wed, 11 Mar 2015 21:05:55 GMT
(Apologies for hitting send too soon and more vendor specific info.)

Liam, FYI CM 5.2.z running CDH5.3.z isn't a supported configuration[1] and
might be the source of your problem.

[1]:
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/pcm_cdh_cm.html

On Wed, Mar 11, 2015 at 4:02 PM, Sean Busbey <busbey@cloudera.com> wrote:

> Thanks for the follow up Liam! I'll add this as a bug against CDH.
>
> On Wed, Mar 11, 2015 at 3:58 PM, Liam Slusser <lslusser@gmail.com> wrote:
>
>> I just wanted to update this thread as after more investigation I've
>> figured out why my oldWALs folder wasn't being cleaned up.  I had a look
>> at
>> the code to ReplicationLogCleaner and it makes this call:
>>
>>  if (!config.getBoolean(HConstants.REPLICATION_ENABLE_KEY,
>>         HConstants.REPLICATION_ENABLE_DEFAULT)) {
>>       LOG.warn("Not configured - allowing all wals to be deleted");
>>       return;
>>     }
>>
>> I searched through my logs and was never able to find that line of text.
>> So wrote a quick program to run that piece of code, and sure enough it
>> came
>> back as True.  getBoolean returns the value if it's been defined and if
>> not
>> returns the default.  And after reading HBASE-3489 replication is enabled
>> by default these days, which I also verified by looking at
>> HConstants.REPLICATION_ENABLE_DEFAULT.  I run cloudera CDH5.3 and in the
>> user interface even with hbase replication set to false, wasn't putting
>> "hbase.replication" false in the configuration file.  I manually added the
>> hbase.replication to false in the advance configuration for hbase-site.xml
>> and restarted hbase and sure enough it deleted all the logs!
>>
>> So this is probably a bug in CDH, at least in the version that I ran.  I'm
>> running cloudera manger 5.2.1 with CDH5.3.0-1.cdh5.3.0.p0.30.
>>
>> thanks,
>> liam
>>
>>
>> On Wed, Mar 4, 2015 at 5:18 PM, Liam Slusser <lslusser@gmail.com> wrote:
>>
>> > So after removing all the replication peers hbase still doesn't want to
>> > clean up the oldWALs folder.  In the master logs I don't see any errors
>> > from ReplicationLogCleaner or LogCleaner.  I have my logging set to
>> INFO so
>> > I'd think I would see something.
>> >
>> > Is there anyway to run the ReplicationLogCleaner manually and see the
>> > output?  Can I write something that calls the right API functions?
>> >
>> > thanks,
>> > liam
>> >
>> >
>> > On Fri, Feb 27, 2015 at 1:50 PM, Nick Dimiduk <ndimiduk@gmail.com>
>> wrote:
>> >
>> >> I would let the cleaner chore handle the cleanup for you. You don't
>> know
>> >> the state of all entries in that folder. To that extent, I'd avoid
>> making
>> >> any direct changes to the content of HBase's working directory,
>> especially
>> >> while HBase is running...
>> >>
>> >> On Fri, Feb 27, 2015 at 1:29 PM, Liam Slusser <lslusser@gmail.com>
>> wrote:
>> >>
>> >> > Once I disable/remove the replication, can I just blow away the
>> oldWALs
>> >> > folder safely?
>> >> >
>> >> > On Fri, Feb 27, 2015 at 3:10 AM, Madeleine Piffaretti <
>> >> > mpiffaretti@powerspace.com> wrote:
>> >> >
>> >> > > Thanks a lot!
>> >> > >
>> >> > > Indeed, we had a replication enable in the past because we used
the
>> >> > > hbase-indexer from NgData (use to replicate data from Hbase to
>> Solr).
>> >> > > The replication was disable from a long time but the hbase-indexer
>> >> peer
>> >> > was
>> >> > > still activated and so, as you mentioned, the data was keept 
to
>> >> > guarantee
>> >> > > to not lose data between disable and enable.
>> >> > >
>> >> > > I have removed the peer and empty the oldWALs folder.
>> >> > >
>> >> > >
>> >> > >
>> >> > > 2015-02-27 1:42 GMT+01:00 Liam Slusser <lslusser@gmail.com>:
>> >> > >
>> >> > > > Huge thanks, Enis, that was the information I was looking
for.
>> >> > > >
>> >> > > > Cheers!
>> >> > > > liam
>> >> > > >
>> >> > > >
>> >> > > > On Thu, Feb 26, 2015 at 3:48 PM, Enis Söztutar <
>> enis.soz@gmail.com>
>> >> > > wrote:
>> >> > > >
>> >> > > > > @Madeleine,
>> >> > > > >
>> >> > > > > The folder gets cleaned regularly by a chore in master.
When a
>> WAL
>> >> > file
>> >> > > > is
>> >> > > > > not needed any more for recovery purposes (when HBase
can
>> guaratee
>> >> > > HBase
>> >> > > > > has flushed all the data in the WAL file), it is moved
to the
>> >> oldWALs
>> >> > > > > folder for archival. The log stays there until all other
>> >> references
>> >> > to
>> >> > > > the
>> >> > > > > WAL file are finished. There is currently two services
which
>> may
>> >> keep
>> >> > > the
>> >> > > > > files in the archive dir. First is a TTL process, which
ensures
>> >> that
>> >> > > the
>> >> > > > > WAL files are kept at least for 10 min. This is mainly
for
>> >> debugging.
>> >> > > You
>> >> > > > > can reduce this time by setting hbase.master.logcleaner.ttl
>> >> > > configuration
>> >> > > > > property in master. It is by default 600000. The other
one is
>> >> > > > replication.
>> >> > > > > If you have replication setup, the replication processes
will
>> >> hang on
>> >> > > to
>> >> > > > > the WAL files until they are replicated. Even if you
disabled
>> the
>> >> > > > > replication, the files are still referenced.
>> >> > > > >
>> >> > > > > You can look at the logs from master from classes (LogCleaner,
>> >> > > > > TimeToLiveLogCleaner, ReplicationLogCleaner) to see
whether the
>> >> > master
>> >> > > is
>> >> > > > > actually running this chore and whether it is getting
any
>> >> exceptions.
>> >> > > > >
>> >> > > > > @Liam,
>> >> > > > > Disabled replication will still hold on to the WAL files
>> because,
>> >> > > because
>> >> > > > > it has a guarantee to not lose data between disable
and enable.
>> >> You
>> >> > can
>> >> > > > > remove_peer, which frees up the WAL files to be eligible
for
>> >> > deletion.
>> >> > > > When
>> >> > > > > you re-add replication peer again, the replication will
start
>> from
>> >> > the
>> >> > > > > current status, versus if you re-enable a peer, it will
>> continue
>> >> from
>> >> > > > where
>> >> > > > > it left.
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > On Thu, Feb 26, 2015 at 12:56 AM, Madeleine Piffaretti
<
>> >> > > > > mpiffaretti@powerspace.com> wrote:
>> >> > > > >
>> >> > > > > > Hi,
>> >> > > > > >
>> >> > > > > > The replication is not turned on HBase...
>> >> > > > > > Does this folder should be clean regularly? Because
I have
>> data
>> >> > from
>> >> > > > > > december 2014...
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > 2015-02-26 1:40 GMT+01:00 Liam Slusser <lslusser@gmail.com>:
>> >> > > > > >
>> >> > > > > > > I'm having this same problem.  I had replication
enabled
>> but
>> >> have
>> >> > > > since
>> >> > > > > > > been disabled.  However oldWALs still grows.
 There are so
>> >> many
>> >> > > files
>> >> > > > > in
>> >> > > > > > > there that running "hadoop fs -ls /hbase/oldWALs"
runs out
>> of
>> >> > > memory.
>> >> > > > > > >
>> >> > > > > > > On Wed, Feb 25, 2015 at 9:27 AM, Nishanth
S <
>> >> > > nishanth.2884@gmail.com
>> >> > > > >
>> >> > > > > > > wrote:
>> >> > > > > > >
>> >> > > > > > > > Do you have replication turned on in
hbase and  if so is
>> >> your
>> >> > > slave
>> >> > > > > > > >  consuming the replicated data?.
>> >> > > > > > > >
>> >> > > > > > > > -Nishanth
>> >> > > > > > > >
>> >> > > > > > > > On Wed, Feb 25, 2015 at 10:19 AM, Madeleine
Piffaretti <
>> >> > > > > > > > mpiffaretti@powerspace.com> wrote:
>> >> > > > > > > >
>> >> > > > > > > > > Hi all,
>> >> > > > > > > > >
>> >> > > > > > > > > We are running out of space in our
small hadoop cluster
>> >> so I
>> >> > > was
>> >> > > > > > > checking
>> >> > > > > > > > > disk usage on HDFS and I saw that
most of the space was
>> >> > > occupied
>> >> > > > by
>> >> > > > > > > the*
>> >> > > > > > > > > /hbase/oldWALs* folder.
>> >> > > > > > > > >
>> >> > > > > > > > > I have checked in the "HBase Definitive
Book" and
>> others
>> >> > books,
>> >> > > > > > > web-site
>> >> > > > > > > > > and I have also search my issue
on google but I didn't
>> >> find a
>> >> > > > > proper
>> >> > > > > > > > > response...
>> >> > > > > > > > >
>> >> > > > > > > > > So I would like to know what does
this folder, what is
>> use
>> >> > for
>> >> > > > and
>> >> > > > > > also
>> >> > > > > > > > how
>> >> > > > > > > > > can I free space from this folder
without breaking
>> >> > > everything...
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > If it's related to a specific version...
our cluster is
>> >> under
>> >> > > > > > > > > 5.3.0-1.cdh5.3.0.p0.30 from cloudera
(hbase 0.98.6).
>> >> > > > > > > > >
>> >> > > > > > > > > Thx for your help!
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
>
>
> --
> Sean
>



-- 
Sean

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message