hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vladrodio...@gmail.com>
Subject Re: [jira] [Resolved] (HBASE-14036) HBase Backup/Restore Phase 1: Custom WAL archive cleaner
Date Sun, 19 Jul 2015 00:03:06 GMT
I agree that it could be done more efficiently. Instead of sending files
one-by-one, plugins
could process WAL archive directory content (all files). For very large
clusters this would save
a lot of time for BackupLogCleaner, but because it is  a background thread,
the performance benefit is not clear
for client applications. If we run cleaners once a day, it does not matter
will they finish in 10 min or in 10 hours.

-Vlad

On Fri, Jul 17, 2015 at 11:38 PM, rahul gidwani <rahul.gidwani@gmail.com>
wrote:

> Do you think its time to stop sending files one by one to the cleaners and
> have all the cleaners take batches of files (like replication and snapshot
> cleaners)?  As we add more cleaners for large clusters and certain
> scenarios the cleaner chain comes to a crawl?
>
> On Fri, Jul 17, 2015 at 3:28 PM, Vladimir Rodionov (JIRA) <jira@apache.org
> >
> wrote:
>
> >
> >      [
> >
> https://issues.apache.org/jira/browse/HBASE-14036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> > ]
> >
> > Vladimir Rodionov resolved HBASE-14036.
> > ---------------------------------------
> >     Resolution: Implemented
> >
> > This feature is part of patch v4. See parent JIRA.
> >
> > > HBase Backup/Restore Phase 1: Custom WAL archive cleaner
> > > --------------------------------------------------------
> > >
> > >                 Key: HBASE-14036
> > >                 URL: https://issues.apache.org/jira/browse/HBASE-14036
> > >             Project: HBase
> > >          Issue Type: Task
> > >            Reporter: Vladimir Rodionov
> > >            Assignee: Vladimir Rodionov
> > >             Fix For: 2.0.0
> > >
> > >
> > > Custom WAL archive cleaner (BackupLogCleaner).  We need to keep WAL
> > files in archive until they either get copied over to backup destination
> > during an incremental backup or full backup (for ALL tables) happens.
> This
> > is tricky, but is doable. Backup-aware WAL archiver cleaner should
> consult
> > hbase:backup to determine if WAL file is safe to purge.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.3.4#6332)
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message