hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Becker <_martinbec...@web.de>
Subject Re: Passing messages
Date Mon, 20 Dec 2010 08:57:47 GMT
Thank you for your suggestions. In this context I heard about
ZooKeeper a few times. It seems to be the easiest and most failsafe
solution as of yet. Another solution mentioned was using some sort of
communication through the file system, which is probably slow and is
quite subtle. Of course, I would like to see a Hadoop/MapReduce
internal solution. I will give the counters a try, first. If that does
not work I will use ZooKeeper instead. All the while hoping for any
Hadoop/MapReduce internal solution to come up.

On Mon, Dec 20, 2010 at 4:24 AM, David Rosenstrauch <darose@darose.net> wrote:
> On 12/18/2010 11:04 AM, Martin Becker wrote:
>>
>> Hello everbody,
>>
>> I am wondering if there is a feature allowing (in my case) reduce
>> tasks to communicate. For example by some volatile variables at some
>> centralized point. Or maybe just notify other running or to-be-running
>> reduce tasks of a completed reduce task featuring some arguments.
>>
>> In my case, I have reduce tasks doing computations that will
>> output/produce certain quality threshold. Other reduce tasks can/could
>> estimate, if they ever get above those thresholds. If not they could
>> just cease running.
>>
>> Thanks in advance,
>> Martin
>>
>> PS: If such functions are not yet part of the API, I would like to
>> know if there are good reasons for it and if not, propose to introduce
>> such functionality.
>
> You could use Zookeeper to do this.
>
> DR
>

Mime
View raw message