hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rares Vernica <rvern...@gmail.com>
Subject performance vs. number of maps
Date Fri, 17 Jul 2009 02:31:50 GMT

I understand that mapred.map.tasks is just a recommendation for the
framework. I also know that, by default, one map task takes as input
one block of data, so the lower limit to the number of maps is equal
to the number of blocks of input data. (There is also the
one-map-per-file constraint.) But, using the default InputFormat, I
can bypass the one-block-per-map limit by setting the
mapred.min.split.size variable. So, basically, one can control the
number of maps very precisely.

>From a performance standpoint (without considering failures), what
would be a good number of maps per node/core? If I want to finish the
task as soon as possible and I have a fixed number of nodes/cores, how
many maps should I assign per node/core? If, for example, I need the
same amount of time to process any of the records, then one map per
core might be a good choice as I would minimize the task startup time.
If different records have different processing times, then 10 maps per
core might be a good choice as, if I have 2 cores per node and one
core can finish its maps earlier it can process maps from the other
core. Overall, I would have at most 10% idle time on cores. If I
increase the number of maps, task start-up time might start to take a
large fraction of the total time, so I don't want to go there either.

Are there any Hadoop insides that would help make a better decision on
the number of maps? For example, how much time does it take to start a


View raw message