hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Akira AJISAKA (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6607) Enable regex pattern matching when mapreduce.task.files.preserve.filepattern is set
Date Sun, 22 May 2016 22:05:12 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Akira AJISAKA updated MAPREDUCE-6607:
-------------------------------------
    Release Note: Before this fix, the files in .staging directory are always preserved when
mapreduce.task.files.preserve.filepattern is set. After this fix, the files in .staging directory
are preserved if their names match the regex pattern specified by mapreduce.task.files.preserve.filepattern.
         Summary: Enable regex pattern matching when mapreduce.task.files.preserve.filepattern
is set  (was: .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or
mapreduce.task.files.preserve.filepattern are set)

> Enable regex pattern matching when mapreduce.task.files.preserve.filepattern is set
> -----------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6607
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6607
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 2.7.1
>            Reporter: Maysam Yabandeh
>            Assignee: Kai Sasaki
>            Priority: Minor
>         Attachments: MAPREDUCE-6607-branch-2.01.patch, MAPREDUCE-6607-branch-2.02.patch,
MAPREDUCE-6607.01.patch, MAPREDUCE-6607.02.patch, MAPREDUCE-6607.03.patch, MAPREDUCE-6607.04.patch,
MAPREDUCE-6607.05.patch, MAPREDUCE-6607.06.patch
>
>
> if either of the following configs are set, then .staging dir is not cleaned up:
> * mapreduce.task.files.preserve.failedtask 
> * mapreduce.task.files.preserve.filepattern
> The former was supposed to keep only .staging of failed tasks and the latter was supposed
to be used only if that task name matches against the specified regular expression.
> {code}
>   protected boolean keepJobFiles(JobConf conf) {
>     return (conf.getKeepTaskFilesPattern() != null || conf
>         .getKeepFailedTaskFiles());
>   }
> {code}
> {code}
>   public void cleanupStagingDir() throws IOException {
>     /* make sure we clean the staging files */
>     String jobTempDir = null;
>     FileSystem fs = getFileSystem(getConfig());
>     try {
>       if (!keepJobFiles(new JobConf(getConfig()))) {
>         jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR);
>         if (jobTempDir == null) {
>           LOG.warn("Job Staging directory is null");
>           return;
>         }
>         Path jobTempDirPath = new Path(jobTempDir);
>         LOG.info("Deleting staging directory " + FileSystem.getDefaultUri(getConfig())
+
>             " " + jobTempDir);
>         fs.delete(jobTempDirPath, true);
>       }
>     } catch(IOException io) {
>       LOG.error("Failed to cleanup staging dir " + jobTempDir, io);
>     }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message