Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-user@lucene.apache.org
Received-SPF: neutral (asf.osuosl.org: local policy)
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: Map Tasks assignment to nodes
Date: Thu, 11 May 2006 17:45:56 -0700
Message-ID: 
 <9EAEB8710A398D4D960257830756F5780616902C@exchg02-bur.search.corpsys.p4pnet.net>
Thread-Topic: Map Tasks assignment to nodes
Thread-Index: AcZ1XW8XEiEsKBU+SZ+DLlpoOgTY6w==
From: "Vijay Murthi" <murthij@yahoo-inc.com>
To: <hadoop-user@lucene.apache.org>

I am trying to run 8 map tasks with 2 reduce on 3 machines. Each task
runs on a 6 MB text file and 500 such files. The monitoring page shows
very few number of Map tasks running than intended. Sometimes some nodes
doesn't even get any tasks assigned though there are large number of
files remaining needs to be scheduled for map operation. Is it due to
distributing the files across nodes? In fact, my file system is set to
local.

Some important parameters are listed below
Io.sort.factor=3D100
Io.sort.mb  =3D 1000
Io.file.buffer.size =3D 4096000
Io.bytes.checksum=3D128

Mapred.map.tasks=3D16
Mapred.reduce.tasks=3D2
Mapred.tasktracker.tasks.maximum=3D4
Mapred.combine.buffer.size=3D100000


Is there any parameter I am missing to maximize the use of all CPUS?=20


Thanks,
VJ