hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kartashov, Andy" <Andy.Kartas...@mpac.ca>
Subject RE: Map Reduce slot
Date Thu, 01 Nov 2012 20:22:08 GMT
I checked my MR admin page running on <host>:50030:jobtracker.jsp this is what I am learning.

I run 2-core processor, so the admin page told me   that my max map/reduce slot capacity was
 14 or so I assume 7 nodes x 2slots.

I did not touch the property of .map.tasks. It seemed that MR set it nicely to 14 automatically,
processing 45 of my maps.  However, I increased reduce task from the default 1 to 14 in the
mapred-site.xml of the JobTracker.


The job worked really fast.

Running Map TasksRunning Reduce TasksTotal SubmissionsNodesOccupied Map SlotsOccupied Reduce
SlotsReserved Map SlotsReserved Reduce SlotsMap Task CapacityReduce Task CapacityAvg. Tasks/NodeBlacklisted
NodesExcluded Nodes14017<http://ec2-54-242-185-163.compute-1.amazonaws.com:50030/machines.jsp?type=active>1400014144.000<http://ec2-54-242-185-163.compute-1.amazonaws.com:50030/machines.jsp?type=blacklisted>1<http://ec2-54-242-185-163.compute-1.amazonaws.com:50030/machines.jsp?type=excluded>
From: kapil bhosale [mailto:kapils.07@gmail.com]
Sent: Thursday, November 01, 2012 3:51 AM
To: user@hadoop.apache.org
Subject: Re: Map Reduce slot

Hi Uddipan,

In my Opinion
 Hadoop Jobtracker Interface provides every Details Including
   # number of Map slots
   # number of Reducer slots

 and other Infromation can also be found in Log files generated.
 Which can also be explored through the same Interface.
 Jobtracker address : Masternode:your_port/jobtracker.jsp<http://>

Other useful details can also be found.


On Thu, Nov 1, 2012 at 1:11 PM, Uddipan Mukherjee <Uddipan_Mukherjee@infosys.com<mailto:Uddipan_Mukherjee@infosys.com>>
Hi Hadoop Gurus.

  What is the good way to know following information in my Hadoop cluster.

# number of Map slots
# number of Reducer slots
JVM Heapsize for Map task
JVM heapsize for reduce task
reuse JVM flag

Thanks and Regards

**************** CAUTION - Disclaimer *****************
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
for the use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you are not
to copy, disclose, or distribute this e-mail or its contents to any other person and
any such actions are unlawful. This e-mail may contain viruses. Infosys has taken
every reasonable precaution to minimize this risk, but is not liable for any damage
you may sustain as a result of any virus in this e-mail. You should carry out your
own virus checks before opening the e-mail or attachment. Infosys reserves the
right to monitor and review the content of all messages sent to or from this e-mail
address. Messages sent to or from this e-mail address may be stored on the
Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and
may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not
the intended recipient, please delete and contact the sender immediately. Please consider
the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe
qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts
par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite.
Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement
l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel

View raw message