hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amareshwari Sriramadasu <amar...@yahoo-inc.com>
Subject Re: OutputCommitter for rollbacks?
Date Mon, 10 Aug 2009 04:05:26 GMT
Hi Deepika,

You can use the fact that map progress and reduce progress 1.0 for 
succeeded jobs and is <1.0 for failed or killed jobs.
Hope this helps.

Thanks
Amareshwari

Deepika Khera wrote:
> Hi,
>
>  
>
> I am trying to use the OutputCommitter.cleanupJob() to commit and
> rollback my job. The cleanup() method is called whether the job was
> successful, killed or failed.
>
>  
>
> I need to know in the cleanupJob(jobContext) method the status of job so
> far, especially if it failed or was killed.
>
>  
>
> The issue is that no matter whether the job failed or was
> killed/successful, the job status in the committer is "running" (which
> makes sense but not what I need). It seems that I can only know the job
> status when the entire job (including the cleanup task) has finished. 
>
>  
>
> I see an open JIRA  which is related , but until we have a resolution to
> that is there any other  way to achieve this?
>
>  
>
> http://issues.apache.org/jira/browse/HADOOP-6005
>
>  
>
> So, what I need is make the decision in the cleanupJob() whether I
> should do  a commit or rollback(Killed/Failed vs Successful).
>
>  
>
> Would appreciate any help on this.
>
>  
>
> Thanks,
>
> Deepika
>
>
>   


Mime
View raw message