hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From abhinav gupta <abhinav_...@yahoo.com>
Subject Abort a job when a counter reaches to a threshold
Date Fri, 24 May 2013 07:43:32 GMT
Hi,

While running a map-reduce job, that has only mappers, I have a counter that counts the number
of failed documents .And after all the 
mappers are done, I want the job to fail if the total number of failed 
documents are above a fixed fraction. ( I need it in the end because I 
don't know the total number of documents initially). How can I achieve 
this without implementing a reduce just for this ?
I know that there are task level cleanup method. But is there any job level cleanup method,
that can be used to perform this after all the 
tasks are done ?

Thanks
Abhinav

Mime
View raw message