hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hwang <joe.haiw...@gmail.com>
Subject how to restrict the concurrent running map tasks?
Date Fri, 18 Jan 2013 10:22:46 GMT
Hi all:

My hadoop version is 1.0.2. Now I want at most 10 map tasks running at the
same time. I have found 2 parameter related to this question.

a) mapred.job.map.capacity

but in my hadoop version, this parameter seems abandoned.

b) mapred.jobtracker.taskScheduler.maxRunningTasksPerJob (
http://grepcode.com/file/repo1.maven.org/maven2/com.ning/metrics.collector/1.0.2/mapred-default.xml
)

I set this variable like below:

Configuration conf = new Configuration();
conf.set("date", date);
conf.set("mapred.job.queue.name", "hadoop");
conf.set("mapred.jobtracker.taskScheduler.maxRunningTasksPerJob", "10");

DistributedCache.createSymlink(conf);
Job job = new Job(conf, "ConstructApkDownload_" + date);
...

The problem is that it doesn't work. There is still more than 50 maps
running as the job starts.

I'm not sure whether I set this parameter in wrong way ? or misunderstand
it.

After looking through the hadoop document, I can't find another parameter
to limit the concurrent running map tasks.

Hope someone can help me ,Thanks.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message