Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4FA7EE2A1 for ; Fri, 18 Jan 2013 11:45:37 +0000 (UTC) Received: (qmail 96529 invoked by uid 500); 18 Jan 2013 11:45:32 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 93373 invoked by uid 500); 18 Jan 2013 11:44:57 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 93343 invoked by uid 99); 18 Jan 2013 11:44:56 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jan 2013 11:44:56 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates 209.85.223.180 as permitted sender) Received: from [209.85.223.180] (HELO mail-ie0-f180.google.com) (209.85.223.180) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jan 2013 11:44:50 +0000 Received: by mail-ie0-f180.google.com with SMTP id c10so6051447ieb.25 for ; Fri, 18 Jan 2013 03:44:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:content-type:x-gm-message-state; bh=DpTs+3W468Om+5TSqq2TA1gqCIk7V2XH2KbRksNluUo=; b=P191pD0h3bDjRX7je4rAu2WGllXwPGHSfxb6c16JngK+Yv0ZRPPYdnZR17+BdlteSb uKlpk/dbQB1GDR0eNYmfuimmCOm6rlIH9vd+hMWPldFJeMvVtDuS06v/fIty0tkbeb8D +vOJLv+fbXH4OHoX9T5CWc/Gsm/jJfozPhSnouvyudAh1G7dKUCaD570XBXei9K5zl7n RnZFiC4xlC5ikmueyI308qYE5EFk8eJUTVrl9AEQux2yDyBDSUff/OFuyOCEnWuiL4rd tdiGZmih18PWZSuBrpYpC3wGJd4CLyWf58+nOW7Ef6hiw901Z/eRaBEO+8QOIk/2yX2j wU9g== X-Received: by 10.42.48.147 with SMTP id s19mr5689744icf.18.1358509469245; Fri, 18 Jan 2013 03:44:29 -0800 (PST) MIME-Version: 1.0 Received: by 10.64.32.166 with HTTP; Fri, 18 Jan 2013 03:44:09 -0800 (PST) In-Reply-To: References: From: Harsh J Date: Fri, 18 Jan 2013 17:14:09 +0530 Message-ID: Subject: Re: how to restrict the concurrent running map tasks? To: joe.haiwang@gmail.com, "" Content-Type: multipart/alternative; boundary=90e6ba1efbc256a27004d38ea26b X-Gm-Message-State: ALoCoQmXMnO5VmEx7TAhvTkV36B1szWnTfoezqXKAl+53AP8HM948XzdKSTWKD+cvoidWq3PrOJe X-Virus-Checked: Checked by ClamAV on apache.org --90e6ba1efbc256a27004d38ea26b Content-Type: text/plain; charset=ISO-8859-1 You will need to use an alternative scheduler for this. Look at minMaps/maxMaps/etc. properties in FairScheduler at http://hadoop.apache.org/docs/stable/fair_scheduler.html#Allocation+File+%28fair-scheduler.xml%29 Alternatively, look at resource-based scheduling in CapacityScheduler at http://hadoop.apache.org/docs/stable/capacity_scheduler.html#Resource+based+scheduling P.s. Do not use general@ list for user level queries. The right list is user@hadoop.apache.org. On Fri, Jan 18, 2013 at 3:52 PM, hwang wrote: > Hi all: > > My hadoop version is 1.0.2. Now I want at most 10 map tasks running at the > same time. I have found 2 parameter related to this question. > > a) mapred.job.map.capacity > > but in my hadoop version, this parameter seems abandoned. > > b) mapred.jobtracker.taskScheduler.maxRunningTasksPerJob ( > > http://grepcode.com/file/repo1.maven.org/maven2/com.ning/metrics.collector/1.0.2/mapred-default.xml > ) > > I set this variable like below: > > Configuration conf = new Configuration(); > conf.set("date", date); > conf.set("mapred.job.queue.name", "hadoop"); > conf.set("mapred.jobtracker.taskScheduler.maxRunningTasksPerJob", "10"); > > DistributedCache.createSymlink(conf); > Job job = new Job(conf, "ConstructApkDownload_" + date); > ... > > The problem is that it doesn't work. There is still more than 50 maps > running as the job starts. > > I'm not sure whether I set this parameter in wrong way ? or misunderstand > it. > > After looking through the hadoop document, I can't find another parameter > to limit the concurrent running map tasks. > > Hope someone can help me ,Thanks. > -- Harsh J --90e6ba1efbc256a27004d38ea26b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
You will need to use an alternative scheduler for this.
Look at minMaps/maxMaps/etc. properties in FairSched= uler at=A0http://hadoop.apache.org/docs/= stable/fair_scheduler.html#Allocation+File+%28fair-scheduler.xml%29
Alternatively, look at resource-based scheduling in CapacitySche= duler at=A0http://hadoop.apache.org/docs/stable/cap= acity_scheduler.html#Resource+based+scheduling

P.s. Do not use general@ list for user leve= l queries. The right list is user= @hadoop.apache.org.


On Fri, Jan 18, 2013 at 3:52 PM, hwang <joe.haiwang@gmail.com><= /span> wrote:
Hi all:

My hadoop version is 1.0.2. Now I want at most 10 map tasks running at the<= br> same time. I have found 2 parameter related to this question.

a) mapred.job.map.capacity

but in my hadoop version, this parameter seems abandoned.

b) mapred.jobtracker.taskScheduler.maxRunningTasksPerJob (
http://grepcode.com/= file/repo1.maven.org/maven2/com.ning/metrics.collector/1.0.2/mapred-default= .xml
)

I set this variable like below:

Configuration conf =3D new Configuration();
conf.set("date", date);
conf.set("m= apred.job.queue.name", "hadoop");
conf.set("mapred.jobtracker.taskScheduler.maxRunningTasksPerJob",= "10");

DistributedCache.createSymlink(conf);
Job job =3D new Job(conf, "ConstructApkDownload_" + date);
...

The problem is that it doesn't work. There is still more than 50 maps running as the job starts.

I'm not sure whether I set this parameter in wrong way ? or misundersta= nd
it.

After looking through the hadoop document, I can't find another paramet= er
to limit the concurrent running map tasks.

Hope someone can help me ,Thanks.



--
Harsh J
--90e6ba1efbc256a27004d38ea26b--