hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-11366) Clean up old .ckpt files after saveNamespace
Date Tue, 24 Jan 2017 22:50:26 GMT

     [ https://issues.apache.org/jira/browse/HDFS-11366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Xiao Chen resolved HDFS-11366.
    Resolution: Duplicate

Looks like we already have HDFS-3716 in place to take care of this problem. Sorry didn't find
that earlier.

It's more aggressive than proposed here, but since the purge only happens after a successful
checkpoint, the risk is low.

> Clean up old .ckpt files after saveNamespace
> --------------------------------------------
>                 Key: HDFS-11366
>                 URL: https://issues.apache.org/jira/browse/HDFS-11366
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs, namenode
>    Affects Versions: 2.6.0
>            Reporter: Xiao Chen
>            Assignee: Xiao Chen
> Checkpoints are done in the NN by writing to {{fsimage.ckpt_TXID}} files, and rename
to {{fsimage_TXID}} files upon success.
> If a checkpoint fails half way, the fsimage.ckpt_ file will be left on disk. There is
no logic to clean it up at all.
> After talking with [~atm], I understand the historical reason for not immediately cleaning
up those files, since they maybe useful for disaster recovery.
> But feels like cleaning those ckpt files after a successful checkpoint, with a larger
TXID threshold is also safe to do.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

View raw message