hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Deepika Khera" <deepi...@collarity.com>
Subject RE: OutputCommitter for rollbacks?
Date Mon, 10 Aug 2009 19:37:51 GMT
Thanks Amareshwari for your response.

It seems like a good idea to use the map progress & reduce progress. My
only concern is that in the web interface(jobdetails.jsp) , we see some
of our jobs show 100% map & 100% reduce, while the reduce still seems to
be running(Not sure but maybe it's just a UI thing). 

But I guess if the job has reached its cleanup, we should be able to
trust these numbers(map progress & reduce progress) and make the call on
commit & rollback?

Thanks again,
Deepika


-----Original Message-----
From: Amareshwari Sriramadasu [mailto:amarsri@yahoo-inc.com] 
Sent: Sunday, August 09, 2009 9:05 PM
To: common-user@hadoop.apache.org
Subject: Re: OutputCommitter for rollbacks?

Hi Deepika,

You can use the fact that map progress and reduce progress 1.0 for 
succeeded jobs and is <1.0 for failed or killed jobs.
Hope this helps.

Thanks
Amareshwari

Deepika Khera wrote:
> Hi,
>
>  
>
> I am trying to use the OutputCommitter.cleanupJob() to commit and
> rollback my job. The cleanup() method is called whether the job was
> successful, killed or failed.
>
>  
>
> I need to know in the cleanupJob(jobContext) method the status of job
so
> far, especially if it failed or was killed.
>
>  
>
> The issue is that no matter whether the job failed or was
> killed/successful, the job status in the committer is "running" (which
> makes sense but not what I need). It seems that I can only know the
job
> status when the entire job (including the cleanup task) has finished. 
>
>  
>
> I see an open JIRA  which is related , but until we have a resolution
to
> that is there any other  way to achieve this?
>
>  
>
> http://issues.apache.org/jira/browse/HADOOP-6005
>
>  
>
> So, what I need is make the decision in the cleanupJob() whether I
> should do  a commit or rollback(Killed/Failed vs Successful).
>
>  
>
> Would appreciate any help on this.
>
>  
>
> Thanks,
>
> Deepika
>
>
>   


Mime
View raw message