hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Kerzner <mark.kerz...@shmsoft.com>
Subject Re: mapred.tasktracker.map.tasks.maximum
Date Tue, 13 Nov 2012 15:52:18 GMT
An important distinction, max slots per node is one thing, and total
mappers is something else - it is the total that ran. But now I know where
to look.

thank you,
mark

On Tue, Nov 13, 2012 at 10:26 AM, Kartashov, Andy <Andy.Kartashov@mpac.ca>wrote:

>  Mark,
>
>
>
> The way I understand it…
>
>
>
> # of mappers is calculated by deviding your input by file split size (64Mb
> default).
>
>
>
> So, say your input is 64Gb in size so you will end up with 1000 mappers.
>
>
>
> Slots are a different story. It is number of map tasks processed by a
> single node in parrallel. It is normally calculated by the number of cores
> that node is running on (one per core).
>
>
>
> Rgds,
>
>
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Tuesday, November 13, 2012 10:20 AM
> *To:* user@hadoop.apache.org
> *Subject:* Re: mapred.tasktracker.map.tasks.maximum
>
>
>
> 1.0.1
>
> On Tue, Nov 13, 2012 at 10:18 AM, Serge Blazhiyevskyy <
> Serge.Blazhiyevskyy@nice.com> wrote:
>
> What hadoop version are we talking about?
>
> From: Mark Kerzner <mark.kerzner@shmsoft.com<mailto:
> mark.kerzner@shmsoft.com>>
> Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <
> user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
> Date: Tuesday, November 13, 2012 5:16 PM
> To: Hadoop User <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
> Subject: mapred.tasktracker.map.tasks.maximum
>
>
> Hi,
>
> I have a cluster with 4 nodes and 32 many cores on each. My default value
> for the maximum number of mappers per slot is 1:
>
>   <property>
>     <name>mapred.tasktracker.map.tasks.maximum</name>
>     <!-- see other kb entry about this one. -->
>     <value>1</value>
>     <final>true</final>
>   </property>
> (which I think is wrong).
>
> Howeve, when sizable jobs run, I see 65 mappers working, so it seems that
> it does create more than 1 mapper per node.
>
> Questions: what maximum number of mappers would be appropriate in this
> situation? Is that the right way to set them?
>
> Thank you. Sincerely,
> Mark
>
>
> NOTICE: This e-mail message and any attachments are confidential, subject
> to copyright and may be privileged. Any unauthorized use, copying or
> disclosure is prohibited. If you are not the intended recipient, please
> delete and contact the sender immediately. Please consider the environment
> before printing this e-mail. AVIS : le présent courriel et toute pièce
> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur
> et peuvent être couverts par le secret professionnel. Toute utilisation,
> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement
> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent
> courriel
>

Mime
View raw message