hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kartashov, Andy" <Andy.Kartas...@mpac.ca>
Subject RE: mapred.tasktracker.map.tasks.maximum
Date Tue, 13 Nov 2012 15:26:47 GMT
Mark,

The way I understand it...

# of mappers is calculated by deviding your input by file split size (64Mb default).

So, say your input is 64Gb in size so you will end up with 1000 mappers.

Slots are a different story. It is number of map tasks processed by a single node in parrallel.
It is normally calculated by the number of cores that node is running on (one per core).

Rgds,

From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
Sent: Tuesday, November 13, 2012 10:20 AM
To: user@hadoop.apache.org
Subject: Re: mapred.tasktracker.map.tasks.maximum

1.0.1
On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <Serge.Blazhiyevskyy@nice.com<mailto:Serge.Blazhiyevskyy@nice.com>>
wrote:
What hadoop version are we talking about?

From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:mark.kerzner@shmsoft.com><mailto:mark.kerzner@shmsoft.com<mailto:mark.kerzner@shmsoft.com>>>
Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org><mailto:user@hadoop.apache.org<mailto:user@hadoop.apache.org>>"
<user@hadoop.apache.org<mailto:user@hadoop.apache.org><mailto:user@hadoop.apache.org<mailto:user@hadoop.apache.org>>>
Date: Tuesday, November 13, 2012 5:16 PM
To: Hadoop User <user@hadoop.apache.org<mailto:user@hadoop.apache.org><mailto:user@hadoop.apache.org<mailto:user@hadoop.apache.org>>>
Subject: mapred.tasktracker.map.tasks.maximum

Hi,

I have a cluster with 4 nodes and 32 many cores on each. My default value for the maximum
number of mappers per slot is 1:

  <property>
    <name>mapred.tasktracker.map.tasks.maximum</name>
    <!-- see other kb entry about this one. -->
    <value>1</value>
    <final>true</final>
  </property>
(which I think is wrong).

Howeve, when sizable jobs run, I see 65 mappers working, so it seems that it does create more
than 1 mapper per node.

Questions: what maximum number of mappers would be appropriate in this situation? Is that
the right way to set them?

Thank you. Sincerely,
Mark

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and
may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not
the intended recipient, please delete and contact the sender immediately. Please consider
the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe
qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts
par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite.
Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement
l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel

Mime
View raw message