Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 7 Dec 2016 19:43:59 +0000 (UTC)
From: "Manoj Govindassamy (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13026445.1481138822000.464041.1481139839321@Atlassian.JIRA>
In-Reply-To: <JIRA.13026445.1481138822000@Atlassian.JIRA>
References: <JIRA.13026445.1481138822000@Atlassian.JIRA> <JIRA.13026445.1481138822981@arcas>
Subject: [jira] [Updated] (HDFS-11218) Add option to skip open files during
 HDFS Snapshots
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 07 Dec 2016 19:44:01 -0000


     [ https://issues.apache.org/jira/browse/HDFS-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Manoj Govindassamy updated HDFS-11218:
--------------------------------------
    Description: 
*Problem:* 

When there are files being written and when HDFS Snapshots are taken in parallel,  Snapshots do capture all these files, but these being written files in Snapshots do not have the point-in-time file length captured.

At the time of File close or any other meta data modification operation on that file which was previously open, HDFS reconciles the file length and records the modification in the last taken Snapshot. All the previously taken Snapshots continue to have the same open File with no modification recorded. So, all those previous snapshots end up using the final modification record in the next available snapshot.

*Proposal:*

HDFS Snapshot Design goal was to have O(M) space usage for Snapshots, where M is the number file modifications. So, it would very expensive to record modifications for all the open files in all the snapshots. For applications that do not want to capture incomplete / partial being written binary files in the snapshots, it would be preferable to have an extra option to skip open files. This way, they don't have to worry about restoring inconsistent files from the snapshots. 

{noformat}
hdfs dfs -createSnapshot -skipOpenFiles <snapDir> <snapName>
{noformat}

  was:
Problem: 

When there are files being written and when HDFS Snapshots are taken in parallel,  Snapshots do capture all these files, but these being written files in Snapshots do not have the point-in-time file length captured.

At the time of File close or any other meta data modification operation on that file which was previously open, HDFS reconciles the file length and records the modification in the last taken Snapshot. All the previously taken Snapshots continue to have the same open File with no modification recorded. So, all those previous snapshots end up using the final modification record in the next available snapshot.

Proposal:

HDFS Snapshot Design goal was to have O(M) space usage for Snapshots, where M is the number file modifications. So, it would very expensive to record modifications for all the open files in all the snapshots. For applications that do not want to capture incomplete / partial being written binary files in the snapshots, it would be preferable to have an extra option to skip open files. This way, they don't have to worry about restoring inconsistent files from the snapshots. 

{noformat}
hdfs dfs -createSnapshot -skipOpenFiles <snapDir> <snapName>
{noformat}


> Add option to skip open files during HDFS Snapshots
> ---------------------------------------------------
>
>                 Key: HDFS-11218
>                 URL: https://issues.apache.org/jira/browse/HDFS-11218
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: snapshots
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Manoj Govindassamy
>            Assignee: Manoj Govindassamy
>
> *Problem:* 
> When there are files being written and when HDFS Snapshots are taken in parallel,  Snapshots do capture all these files, but these being written files in Snapshots do not have the point-in-time file length captured.
> At the time of File close or any other meta data modification operation on that file which was previously open, HDFS reconciles the file length and records the modification in the last taken Snapshot. All the previously taken Snapshots continue to have the same open File with no modification recorded. So, all those previous snapshots end up using the final modification record in the next available snapshot.
> *Proposal:*
> HDFS Snapshot Design goal was to have O(M) space usage for Snapshots, where M is the number file modifications. So, it would very expensive to record modifications for all the open files in all the snapshots. For applications that do not want to capture incomplete / partial being written binary files in the snapshots, it would be preferable to have an extra option to skip open files. This way, they don't have to worry about restoring inconsistent files from the snapshots. 
> {noformat}
> hdfs dfs -createSnapshot -skipOpenFiles <snapDir> <snapName>
> {noformat}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org