Return-Path: 
 <user-return-12470-apmail-hadoop-hdfs-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D0A8110E27
	for <apmail-hadoop-hdfs-user-archive@minotaur.apache.org>;
 Wed, 11 Dec 2013 16:22:12 +0000 (UTC)
Received: (qmail 69326 invoked by uid 500); 11 Dec 2013 16:22:06 -0000
Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org
Received: (qmail 69135 invoked by uid 500); 11 Dec 2013 16:22:03 -0000
Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:user@hadoop.apache.org>
List-Id: <user.hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Delivered-To: mailing list user@hadoop.apache.org
Received: (qmail 69101 invoked by uid 99); 11 Dec 2013 16:22:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2013 16:22:02 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mirko.kaempf@gmail.com
 designates 209.85.160.42 as permitted sender)
Received: from [209.85.160.42] (HELO mail-pb0-f42.google.com) (209.85.160.42)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2013 16:21:56 +0000
Received: by mail-pb0-f42.google.com with SMTP id uo5so10327417pbc.29
        for <user@hadoop.apache.org>; Wed, 11 Dec 2013 08:21:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        bh=dJrBUhTulB1Zqay//2mMyJdlCGJq/SrY73gp8oJWOn4=;
        b=hDSJtFDdVf7o7C5h2oTJOD2hcpGgYQrKHDI7iuse9WTUaDAuDniS84FeAwIsWcbKj1
         q75KSLEioOdZASG9xD1RFcxDgXdaMIC1iGUuRMBKbCD6xKT/V4srO/VKylT9+fystFwB
         bp7qoGYx4YXOaJSqQNtwpYspkxMoYc75Wf0YM56653PeFZ3zNFr1XQQ12UxASlCqF8UM
         7OcUqaZSRGfv1wMBqAydvF8Lsli8T3zV0A5ax1flSn2+XAAyiLDVDZS4wCJics4hoEsB
         0zXuGHoEq96MJPRNgS02KFtFuiZDYRIRbxFOeiHbU54BNl1jUYBXyoWx6mdY3XBxB+ez
         INTQ==
X-Received: by 10.68.231.105 with SMTP id tf9mr2810059pbc.4.1386778895197;
 Wed, 11 Dec 2013 08:21:35 -0800 (PST)
MIME-Version: 1.0
Received: by 10.66.197.195 with HTTP; Wed, 11 Dec 2013 08:21:15 -0800 (PST)
In-Reply-To: <CECE5892.4A4FB%idror@akamai.com>
References: <CECE5892.4A4FB%idror@akamai.com>
From: =?ISO-8859-1?Q?Mirko_K=E4mpf?= <mirko.kaempf@gmail.com>
Date: Wed, 11 Dec 2013 16:21:15 +0000
Message-ID: 
 <CALKHTG-Ex1ZTjwUzPiJK9p5fjr7w4+RhTN9LK96HdOxOJO6c5g@mail.gmail.com>
Subject: Re: Why is Hadoop always running just 4 tasks?
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=047d7b33cac86e2d1604ed449fff
X-Virus-Checked: Checked by ClamAV on apache.org

--047d7b33cac86e2d1604ed449fff
Content-Type: text/plain; charset=ISO-8859-1

Hi,

what is the command you execute to submit the job?
Please share also the driver code ....

So we can troubleshoot better.

Best wishes
Mirko




2013/12/11 Dror, Ittay <idror@akamai.com>

> I have a cluster of 4 machines with 24 cores and 7 disks each.
>
> On each node I copied from local a file of 500G. So I have 4 files in hdfs
> with many blocks. My replication factor is 1.
>
> I run a job (a scalding flow) and while there are 96 reducers pending,
> there are only 4 active map tasks.
>
> What am I doing wrong? Below is the configuration
>
> Thanks,
> Ittay
>
> <configuration>
> <property>
>  <name>mapred.job.tracker</name>
>    <value>master:54311</value>
>  </property>
>
> <property>
>   <name>mapred.map.tasks</name>
>   <value>96</value>
> </property>
>
> <property>
>    <name>mapred.reduce.tasks</name>
>    <value>96</value>
>  </property>
>
> <property>
> <name>mapred.local.dir</name>
>
> <value>/hdfs/0/mapred/local,/hdfs/1/mapred/local,/hdfs/2/mapred/local,/hdfs/3/mapred/local,/hdfs/4/mapred/local,/hdfs/5/mapred/local,/hdfs/6/mapred/local,/hdfs/7/mapred/local</value>
>  </property>
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
>  <value>24</value>
> </property>
>
> <property>
>     <name>mapred.tasktracker.reduce.tasks.maximum</name>
>     <value>24</value>
> </property>
> </configuration>
>

--047d7b33cac86e2d1604ed449fff
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi,</div><div>=A0</div><div>what is the command you e=
xecute to submit the job?<br>Please share also the driver code ....</div><d=
iv>=A0</div><div>So we can troubleshoot better.</div><div>=A0</div><div>Bes=
t wishes</div>

<div>Mirko</div><div>=A0</div><div>=A0</div></div><div class=3D"gmail_extra=
"><br><br><div class=3D"gmail_quote">2013/12/11 Dror, Ittay <span dir=3D"lt=
r">&lt;<a href=3D"mailto:idror@akamai.com" target=3D"_blank">idror@akamai.c=
om</a>&gt;</span><br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div style=3D"font-family:Calibri,sans-serif=
;font-size:14px;word-wrap:break-word"><div>I have a cluster of 4 machines w=
ith 24 cores and 7 disks each.</div>

<div><br></div><div>On each node I copied from local a file of 500G. So I h=
ave 4 files in hdfs with many blocks. My replication factor is 1.</div><div=
><br></div><div>I run a job (a scalding flow) and while there are 96 reduce=
rs pending, there are only 4 active map tasks.=A0</div>

<div><br></div><div>What am I doing wrong? Below is the configuration</div>=
<div><br></div><div>Thanks,</div><div>Ittay</div><div><br></div><div><div>&=
lt;configuration&gt;</div><div><span style=3D"white-space:pre-wrap">	</span=
>&lt;property&gt;</div>

<div><span style=3D"white-space:pre-wrap">		</span>&lt;name&gt;mapred.job.t=
racker&lt;/name&gt;</div><div><span style=3D"white-space:pre-wrap">	</span>=
 =A0<span style=3D"white-space:pre-wrap">	</span>&lt;value&gt;master:54311&=
lt;/value&gt;</div>

<div><span style=3D"white-space:pre-wrap">	</span>&lt;/property&gt;</div><d=
iv><br></div><div><span style=3D"white-space:pre-wrap">	</span>&lt;property=
&gt;</div><div><span style=3D"white-space:pre-wrap">		</span> =A0&lt;name&g=
t;mapred.map.tasks&lt;/name&gt;</div>

<div><span style=3D"white-space:pre-wrap">		</span> =A0&lt;value&gt;96&lt;/=
value&gt;</div><div><span style=3D"white-space:pre-wrap">	</span>&lt;/prope=
rty&gt;</div><div><br></div><div><span style=3D"white-space:pre-wrap">	</sp=
an>&lt;property&gt;</div>

<div><span style=3D"white-space:pre-wrap">	</span> =A0<span style=3D"white-=
space:pre-wrap">	</span>&lt;name&gt;mapred.reduce.tasks&lt;/name&gt;</div><=
div><span style=3D"white-space:pre-wrap">	</span> =A0<span style=3D"white-s=
pace:pre-wrap">	</span>&lt;value&gt;96&lt;/value&gt;</div>

<div><span style=3D"white-space:pre-wrap">	</span>&lt;/property&gt;</div><d=
iv><br></div><div><span style=3D"white-space:pre-wrap">	</span>&lt;property=
&gt;</div><div><span style=3D"white-space:pre-wrap">		</span>&lt;name&gt;ma=
pred.local.dir&lt;/name&gt;</div>

<div><span style=3D"white-space:pre-wrap">		</span>&lt;value&gt;/hdfs/0/map=
red/local,/hdfs/1/mapred/local,/hdfs/2/mapred/local,/hdfs/3/mapred/local,/h=
dfs/4/mapred/local,/hdfs/5/mapred/local,/hdfs/6/mapred/local,/hdfs/7/mapred=
/local&lt;/value&gt;</div>

<div><span style=3D"white-space:pre-wrap">	</span>&lt;/property&gt;</div><d=
iv><br></div><div><span style=3D"white-space:pre-wrap">	</span>&lt;property=
&gt;</div><div><span style=3D"white-space:pre-wrap">		</span>&lt;name&gt;ma=
pred.tasktracker.map.tasks.maximum&lt;/name&gt;</div>

<div><span style=3D"white-space:pre-wrap">		</span>&lt;value&gt;24&lt;/valu=
e&gt;</div><div><span style=3D"white-space:pre-wrap">	</span>&lt;/property&=
gt;</div><div><br></div><div><span style=3D"white-space:pre-wrap">	</span>&=
lt;property&gt;</div>

<div>=A0 =A0 <span style=3D"white-space:pre-wrap">		</span>&lt;name&gt;mapr=
ed.tasktracker.reduce.tasks.maximum&lt;/name&gt;</div><div>=A0 =A0 <span st=
yle=3D"white-space:pre-wrap">		</span>&lt;value&gt;24&lt;/value&gt;</div><d=
iv><span style=3D"white-space:pre-wrap">	</span>&lt;/property&gt;</div>

<div>&lt;/configuration&gt;</div></div></div>
</blockquote></div><br></div>

--047d7b33cac86e2d1604ed449fff--