hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: FileOutputCommiter behavior doubts
Date Wed, 23 May 2012 11:18:13 GMT
This is related: https://issues.apache.org/jira/browse/MAPREDUCE-2493

But the real issue is LocalJobRunner does:

OutputCommitter outputCommitter = job.getOutputCommitter();

Which doesn't work with New API cause the OutputCommitter now comes
from within the OutputFormat class. So hence you get the default
FileOutputCommitter returned from configuration, and hence your issue.

Can you file a JIRA for this? We'll have to fix LocalJobRunner to do
things differently for new API usage.

On Wed, May 23, 2012 at 2:53 PM, Subroto <ssanyal@datameer.com> wrote:
> Hi,
> I have an implementation SampleFileOutputCommiter which
> extends org.apache.hadoop.mapred.FileOutputCommitter . The implementation
> has specific code to be executed during cleanupJob() execution.
> When the framework(LocalJobRunner) makes a call to commitJob(), the
> framework never takes care of calling the cleanupJob() instead it
> calls org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitJob()
> which internally calls its own cleanupJob().
> Though the method cleanupJob is deprecated but, still I feel the framework
> should take care of executing it as it is being executed
> from org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.
> Currently the framework is not letting the Jobs written with MRV1 to run
> properly.
> Any thoughts….. ??
> Cheers,
> Subroto Sanyal

Harsh J

View raw message