hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jag123 <jaganr...@yahoo.com>
Subject Re: Performance issues with large map/reduce processes
Date Thu, 27 Dec 2007 18:31:05 GMT

> Can you say a bit more about your processes?  Are they truly parallel maps
> without any shared state?

My map process reads a line of record from the input and explodes it into a
bunch of
key and value pairs. The number of key-value pair is typically small (around
8 on the average) 
and is independent of the size of the dataset. That is, it only depends on
the input record. 

I use a bunch of reporter objects to monitor the performance of the
map/reduce processes,
but I guess that does not affect performance. I do not access the disk etc. 
The map operation is purely CPU based.

> Are you getting a good limit on maximum number of maps and reduces per
> machine?

The number of maps/reduce per machine is set to two by the administrator. 
I am getting around 140 maps from the 70 machine cluster. 

> How are you measuring these times?  Do they include shuffle time as well
> as
> map time?  Do they include time before running?

Yes - the time includes both the shuffle time and map time. The shuffle time
is less than 30 seconds 
even for the largest tasks as I can see that it has started to process the
(although very very slowly).
I can monitor the progress of each map process using the "reporter" objects.

> What happens on the large size problems if you decrease the number of
> maps,
> but keep input size constant?

My map processes run out of Heap space if I decrease the number of map
The maximum number of records that my map process can handle is 250k as my
heap size is only 256m.

> Finally, why do you have so many reduces?  Usually it is good to have at
> most a small multiple of the number of machines in your cluster.

Yes - I agree and thanks for pointing out. In the large setup, I rarely get
past the map 
stage because it takes tens of hours for all the maps to finish.

> On 12/26/07 2:52 PM, "jag123" wrote:
>> Hi,
>> I am running a map/reduce task on a large cluster (70+ machines). I use a
>> single input file, and sufficient number of map/reduce tasks so that each
>> map process gets 250k records. That is, if my  input file contains 1
>> million records, I use 4 map and 4 reduce processes so that each map
>> process
>> gets 250k records.  The maps/reduce usually takes 30 seconds to complete.
>> A strange thing happens when I scale this problem:
>> 1 million records, 4 map + 4 reduce ==> 30 seconds per map process
>> 5 million records, 20 map + 20 reduce ==>  1 minute per map process
>> 50 million records, 200 map + 200 reduce ==>  3 minute per map process
>> 500 million records, 2000 map + 2000 reduces ==> 45 minutes! per map
>> process
>> Note that in all the above cases, the map process performs the same
>> amount
>> of work (250k records).
>> In all the cases, I use a single large input file. Hadoop breaks the file
>> into ~16 MB chunks (about 250k records). Input format is
>> TextInputFormat.class. I cannot think of any reason why this is
>> happening.
>> The task setup in all the above cases takes 30 seconds or so. But then
>> the
>> map process practically crawls. 

View this message in context: http://www.nabble.com/Performance-issues-with-large-map-reduce-processes-tp14507375p14516774.html
Sent from the Hadoop Users mailing list archive at Nabble.com.

View raw message