hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Walter Su (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (HDFS-8714) Folder ModificationTime in Millis Changed When NameNode is restarted
Date Fri, 03 Jul 2015 01:27:04 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Walter Su reassigned HDFS-8714:
-------------------------------

    Assignee: Walter Su

> Folder ModificationTime in Millis Changed When NameNode is restarted
> --------------------------------------------------------------------
>
>                 Key: HDFS-8714
>                 URL: https://issues.apache.org/jira/browse/HDFS-8714
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Chandan Biswas
>            Assignee: Walter Su
>
> *Steps to Produce*
> # Steps need to do in program
> ** Create a folder into HDFS 
> ** Print folder modificationTime in millis
> ** Upload a file or copy a file to this newly created folder
> ** Print file and folder modificationTime in millis
> ** Restart the name node
> ** Print file and folder modificationTime in millis
> # Expected Result
> ** folder modification time should be the file modification time before name node restart
> ** folder modification time should not change after name node restart
> # Actual result
> ** folder modification time is not same with file modification time
> ** folder modification time is changed after name node restart and it's changed to file
modification time
> *Impact of this behavior:* Before task is launched, distributed cache files/folders are
checked for any modification. The checks are done by comparing file/folder modicationTime
in millis. So any job that uses distributed cache has a potential chance of failure if 
> # name node restarts and running tasks are resubmitted or 
> # for e.g among 100 tasks 50 are in queue for run. Now name node restarts
> Here is the sample code I used for testing-
> {code}
> // file creating in hdfs
>         final Path pathToFiles = new Path("/user/vagrant/chandan/test/");
>         fileSystem.mkdirs(pathToFiles);
>         System.out.println("HDFS Folder Modification Time in long Before file copy:"
>                 + fileSystem.getFileStatus(pathToFiles).getModificationTime());
>         FileUtil.copy(fileSystem, new Path("/user/cloudera/test"), fileSystem, pathToFiles,
false, configuration);
>         System.out.println("HDFS File Modification Time in long:"
>                 + fileSystem.getFileStatus(new Path("/user/vagrant/chandan/test/test")).getModificationTime());
>         System.out.println("HDFS Folder Modification Time in long After file copy:"
>                 + fileSystem.getFileStatus(pathToFiles).getModificationTime());
>         for (int i = 0; i < 100; i++) {
>             System.out.println("Normal HDFS Folder Modification Time in long:"
>                     + fileSystem.getFileStatus(pathToFiles).getModificationTime());
>             System.out.println("Normal HDFS File Modification Time in long:"
>                     + fileSystem.getFileStatus(new Path("/user/vagrant/chandan/test/test")).getModificationTime());
>             Thread.sleep(60000 * 2);
>         }
> {code}
> Here is the output -
> {code}
> HDFS Folder Modification Time in long Before file copy:1435868217309
> HDFS File Modification Time in long:1435868217368
> HDFS Folder Modification Time in long After file copy:1435868217353
> Normal HDFS Folder Modification Time in long:1435868217353
> Normal HDFS File Modification Time in long:1435868217368
> Normal HDFS Folder Modification Time in long:1435868217353
> Normal HDFS File Modification Time in long:1435868217368
> Normal HDFS Folder Modification Time in long:1435868217368
> Normal HDFS File Modification Time in long:1435868217368
> {code}
> The last two lines are printed after name node restart.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message