hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley" <...@yahoo-inc.com>
Subject Re: Aborting Map Function
Date Wed, 16 Apr 2008 17:25:04 GMT
On Apr 16, 2008, at 8:28 AM, Chaman Singh Verma wrote:

> I am developing one application with MapReduce and in that whenever  
> some
> MapTask condition is
> met, I would like to broadcast to all other MapTask to abort their  
> work. I
> am not quite sure whether
> such broadcasting functionality currently exist in Hadoop  
> MapReduce. Could
> someone give some
> hints.

This is pretty atypical behavior, but you could have each map look  
for the existence of an hdfs file every 1 minute or so. When the  
condition is true, create the file and your maps will exit in the  
next minute. Except on very large clusters, that wouldn't be too  
expensive...

-- Owen

Mime
View raw message