hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ron Sher <ron.s...@gmail.com>
Subject Staging area isn't cleaned
Date Sun, 27 Jan 2013 15:29:39 GMT

I'm using Hadoop version 2.0.0-cdh4.1.2,
I'm using MRV1 (not yarn).

Whenever I run a job it says it's about to clean the staging area, but it

I tried some hack in which I do the cleaning up myself using the code below.
This works fine while there are no exceptions in the job, but if there are
(for example, when there are no input files to work on) I get the exception
and my cleanup code doesn't run.

Is there some way I can guarantee that the staging area get cleaned?

Thanks for you help,
String stagingDirToRemove = getStagingDir(conf, job);
 Path stagingDir = new Path(stagingDirToRemove);

LOG.info("about to remove " + stagingDirToRemove);

dfs.delete(stagingDir, true);

private static String getStagingDir(Configuration conf, RunningJob job) {
 String stagingRoot = conf.get("mapreduce.jobtracker.staging.root.dir");
String userName = System.getProperty("user.name");
 String jobID = job.getID().toString();
String result = stagingRoot + "/" + userName + "/.staging/" + jobID;

return result;

View raw message