hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <ma...@eecs.berkeley.edu>
Subject Re: Small doubt in MR
Date Sun, 03 Jan 2010 04:15:54 GMT
If you want the code to happen on only one machine, why not run it in your driver program that
submits the MapReduce job?

You could also create a special input record that tells the mapper who gets that record that
it's the chosen one. However, note that that mapper may be run multiple times due hardware
failures.

Matei

On Jan 2, 2010, at 11:08 PM, Mark Kerzner wrote:

> I think you need some kind of semaphore that you can turn on by the first
> reducer. For example, allocating a file in HDFS would work - if you could
> guarantee that it is an atomic operation (create-if-does-not-exist).
> 
> Mark
> 
> On Sat, Jan 2, 2010 at 10:04 PM, bharath v <
> bharathvissapragada1990@gmail.com> wrote:
> 
>> Hi,
>> 
>> I want a particular "section of code" to run only in any "ONE" of the
>> mappers . So I employed the following procedure.
>> 
>> Main-Class
>> {
>> 
>>       public boolean flag = true;
>> 
>>       Map-Class
>>      {
>>            if(flag)
>>           {
>> 
>>               flag=false;
>>              /* section of code */
>>           }
>> 
>> 
>> }
>> 
>> I am running this code on in pseudo-distributed mode and its working fine .
>> I doubt whether this runs correctly in distributed mode because , mappers
>> on
>> other systems have to notified of the changed "flag" .. Any Comments ? If
>> this is wrong , any suggestions on what method I must follow to achieve
>> this
>> functionality in D-mode .
>> 
>> Thanks
>> 


Mime
View raw message