Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2EAC31051B for ; Wed, 11 Dec 2013 19:29:50 +0000 (UTC) Received: (qmail 65423 invoked by uid 500); 11 Dec 2013 19:29:45 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 65187 invoked by uid 500); 11 Dec 2013 19:29:45 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 65180 invoked by uid 99); 11 Dec 2013 19:29:45 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2013 19:29:45 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [72.246.2.14] (HELO prod-mail-xrelay02.akamai.com) (72.246.2.14) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2013 19:29:41 +0000 Received: from prod-mail-xrelay02.akamai.com (localhost [127.0.0.1]) by postfix.imss70 (Postfix) with ESMTP id 8A9A0284EB for ; Wed, 11 Dec 2013 19:28:48 +0000 (GMT) Received: from prod-mail-relay06.akamai.com (prod-mail-relay06.akamai.com [172.17.120.126]) by prod-mail-xrelay02.akamai.com (Postfix) with ESMTP id 7510E28420 for ; Wed, 11 Dec 2013 19:28:48 +0000 (GMT) Received: from usma1ex-cashub.kendall.corp.akamai.com (usma1ex-cashub6.kendall.corp.akamai.com [172.27.105.22]) by prod-mail-relay06.akamai.com (Postfix) with ESMTP id 6AE3D2034 for ; Wed, 11 Dec 2013 19:28:48 +0000 (GMT) Received: from USMBX1.msg.corp.akamai.com ([169.254.1.14]) by USMA1EX-CASHUB6.kendall.corp.akamai.com ([172.27.105.22]) with mapi; Wed, 11 Dec 2013 14:28:47 -0500 From: "Dror, Ittay" To: "user@hadoop.apache.org" Date: Wed, 11 Dec 2013 14:28:46 -0500 Subject: Re: Why is Hadoop always running just 4 tasks? Thread-Topic: Why is Hadoop always running just 4 tasks? Thread-Index: Ac72pzbQG/1QD6w/S8eyXEM4oGIycg== Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.3.9.131030 acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_CECE88C24A51Aidrorakamaicom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CECE88C24A51Aidrorakamaicom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thank you. The command is: hadoop jar /tmp/Algo-0.0.1.jar com.twitter.scalding.Tool com.akamai.Algo --= hdfs --header --input /algo/input{0..3}.gz --output /algo/output Btw, the Hadoop version is 1.2.1 Not sure what driver you are referring to. Regards, Ittay From: Mirko K=E4mpf > Reply-To: "user@hadoop.apache.org" > Date: Wednesday, December 11, 2013 6:21 PM To: "user@hadoop.apache.org" > Subject: Re: Why is Hadoop always running just 4 tasks? Hi, what is the command you execute to submit the job? Please share also the driver code .... So we can troubleshoot better. Best wishes Mirko 2013/12/11 Dror, Ittay > I have a cluster of 4 machines with 24 cores and 7 disks each. On each node I copied from local a file of 500G. So I have 4 files in hdfs = with many blocks. My replication factor is 1. I run a job (a scalding flow) and while there are 96 reducers pending, ther= e are only 4 active map tasks. What am I doing wrong? Below is the configuration Thanks, Ittay mapred.job.tracker master:54311 mapred.map.tasks 96 mapred.reduce.tasks 96 mapred.local.dir /hdfs/0/mapred/local,/hdfs/1/mapred/local,/hdfs/2/mapred/local,/hdfs= /3/mapred/local,/hdfs/4/mapred/local,/hdfs/5/mapred/local,/hdfs/6/mapred/lo= cal,/hdfs/7/mapred/local mapred.tasktracker.map.tasks.maximum 24 mapred.tasktracker.reduce.tasks.maximum 24 --_000_CECE88C24A51Aidrorakamaicom_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Thank you.
The command is:
hadoop jar /tmp/Algo-0.0.1.jar com.tw= itter.scalding.Tool com.akamai.Algo --hdfs --header --input /algo/input{0..= 3}.gz --output /algo/output

Btw, the Hadoop versio= n is 1.2.1

Not sure what driver you are referring = to. 
Regards,
Ittay

From: Mirko K=E4mpf <mirko.kaempf@gmail.com>
Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
D= ate: Wednesday, December 11, 2013 6:21 PM
To: "user@hado= op.apache.org" <user@hadoo= p.apache.org>
Subject: R= e: Why is Hadoop always running just 4 tasks?

Hi,
 
what is the command you execute to submit = the job?
Please share also the driver code ....
 
So we can= troubleshoot better.
 
Best wishes
Mirk= o
 
 
<= br>
2013/12/11 Dror, Ittay = <idror@akamai.com<= /a>>
I have a clust= er of 4 machines with 24 cores and 7 disks each.

O= n each node I copied from local a file of 500G. So I have 4 files in hdfs w= ith many blocks. My replication factor is 1.

I run= a job (a scalding flow) and while there are 96 reducers pending, there are= only 4 active map tasks. 

What am I doing wr= ong? Below is the configuration

Thanks,
= Ittay

<configuration>
<property>
<name>mapred.job.tracker</name>= ;
 <value>master:54311</value>
</property>
=

<propert= y>
 <name&= gt;mapred.map.tasks</name>
 <value>96</value>
</property>

<property>
 <name>mapred.reduce.tasks</name>
 <value>96</value>
</property>

<property>
<name>mapred.local.dir</name>
<value>/hdfs/0/mapred/= local,/hdfs/1/mapred/local,/hdfs/2/mapred/local,/hdfs/3/mapred/local,/hdfs/= 4/mapred/local,/hdfs/5/mapred/local,/hdfs/6/mapred/local,/hdfs/7/mapred/loc= al</value>
<= /property>

<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>24</value><= /div>
</property>

<prope= rty>
    <name>mapred.tasktracker.reduce.tasks.maximum</name>
    <value>24&= lt;/value>
</pr= operty>
</configuration>
<= /div>
 --_000_CECE88C24A51Aidrorakamaicom_--