hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "GOEKE, MATTHEW (AG/1000)" <matthew.go...@monsanto.com>
Subject RE: tasktracker maximum map tasks for a certain job
Date Tue, 21 Jun 2011 17:44:13 GMT
Off the wall thought but it might be possible to do this through rolling your own load manager
using the fair scheduler. I know this is how people have setup custom job distributions based
on current cluster utilization.


From: Jonathan Zukerman [mailto:zukermanjonathan@gmail.com]
Sent: Tuesday, June 21, 2011 12:17 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: tasktracker maximum map tasks for a certain job

That is a bit problematic because I have other jobs running at the same time, most of them
don't care about the number of map tasks per tasktracker.
Is there a way to implement this in in my job project? What is the best way to do it?
On Tue, Jun 21, 2011 at 8:08 PM, Joey Echeverria <joey@cloudera.com<mailto:joey@cloudera.com>>
The only way to do that is to drop the setting down to one and bounce
the TaskTrackers.


On Tue, Jun 21, 2011 at 12:52 PM, Jonathan Zukerman
<zukermanjonathan@gmail.com<mailto:zukermanjonathan@gmail.com>> wrote:
> Hi,
> Is there a way to set the maximum map tasks for all tasktrackers in my
> cluster for a certain job?
> Most of my tasktrackers are configured to handle 4 maps concurrently, and
> most of my jobs don't care where does the map function run. But small part
> of my jobs requires that no two map functions will run from the same IP at
> the same time, i.e., the maximum number of map tasks for each tasktracker
> should be 1 for these jobs.
> Thanks!
> Jonathan.

Joseph Echeverria
Cloudera, Inc.

This e-mail message may contain privileged and/or confidential information, and is intended
to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the
sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail
by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, reading and archival
by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence
of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such
code transmitted by or accompanying
this e-mail or any attachment.

The information contained in this email may be subject to the export control laws and regulations
of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations
issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC).  As a recipient of this information you
are obligated to comply with all
applicable U.S. export laws and regulations.

View raw message