hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kaczmarek, Eric" <eric.kaczma...@intel.com>
Subject RE: Memory based scheduling
Date Tue, 30 Oct 2012 15:58:19 GMT
Someone might correct me if I am wrong, but isn't the number of mappers determined by your
input size and hdfs block size?

Example your input per system is 1Mb, setting your block size to 512k should result in only
2 mappers to execute on that system?

-Eric

From: Marco Z├╝hlke [mailto:mzuehlke@gmail.com]
Sent: Tuesday, October 30, 2012 8:49 AM
To: user@hadoop.apache.org
Subject: Memory based scheduling

Hi,

on our cluster our jobs usually satisfied with less than 2 GB of heap space.
so we have on our 8 GB computers 3 maps maximum and on our 16 GB
computers 4 maps maximum (we only have quad core CPUs and to have
memory left for reducers). This works very well.

But now we have a new kind of jobs. Each mapper requires at lest 4 GB
of heap space.

Is it possible to limit the number of tasks (mapper) per computer to 1 or 2 for
these kinds of jobs ?

Regards,
Marco

Mime
View raw message