hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Segel, Mike" <mse...@navteq.com>
Subject RE: Jobtrackers scalability
Date Wed, 03 Mar 2010 13:14:49 GMT

What exactly do you mean by 'commodity' hardware?

I mean if you have the budget to build out a 10,000 node cloud, don't you think you'd have
the money to beef up your job tracker?

With respect to the job tracker, its more memory intensive than disk, right?
So how many nodes can you run in a cluster where the job tracker has 96GB of memory?
(Which I think is a 'commodity' box maxed out)


-----Original Message-----
From: kartik [mailto:ekartik@huawei.com] 
Sent: Wednesday, March 03, 2010 2:55 AM
To: common-dev@hadoop.apache.org
Subject: RE: Jobtrackers scalability


hai geeks,



The functionality of the job tracker is to divide the job among the
tasktrackers and monitor the progress of the task, and even the job also...


Do we address a situation where we have similar commodity hardware deployed
suppose of 2GB RAM and there are some 10000 nodes and we have to submit a
single batch job ... (That is think of a scenario where we do not have any
high end computer which alone would be configured as a JobTracker )..  Which
requires to coordinate all the 10000 TaskTrackers.. I think this will surely
degrade the performance of the JobTracker.... 


For applications now-a-days it is impractical to think of a similar
scenario, but just to confirm whether the job tracker will run under such


Was just curious to know what would be the limit in terms of RAM size and
CPU speed of a Job Tracker to support such a application.



Thanks and Regards






This e-mail and attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed
above. Any use of the information contained herein in any way (including,
but not limited to, total or partial disclosure, reproduction, or
dissemination) by persons other than the intended recipient's) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!


The information contained in this communication may be CONFIDENTIAL and is intended only for
the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby
notified that any dissemination, distribution, or copying of this communication, or any of
its contents, is strictly prohibited.  If you have received this communication in error, please
notify the sender and delete/destroy the original message and any copy of it from your computer
or paper files.

View raw message