Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of cubicdesign@gmail.com
 designates 209.85.219.210 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type;
        b=e6M+JHzeCHdR/XgmZpg+4J5bGwL07cNfcxINF/j57Ituw009X8yE1qsjqPP7Is3WVr
         BXW64iSQQdoNSQSt8N54YAE0QuWf/kjKwtFC15Yu/fs9irgB5+pNt5gsxlX1fTwlMwoP
         EjCtu41ngNDNqTxFaLpXYnRHiL9NYpkqYsQE8=
Message-ID: <4B0EE3DE.2060001@Gmail.com>
Date: Thu, 26 Nov 2009 21:23:58 +0100
From: CubicDesign <cubicdesign@gmail.com>
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: common-user@hadoop.apache.org
Subject: Re: Processing 10MB files in Hadoop
References: <efbdc9790911260402x56e92768qe117970cf8c27553@mail.gmail.com>
	 <9b0a5b990911260542q16fab58ag4bd6f875d763d0c4@mail.gmail.com>
	 <8211a1320911260557v7b895b05x8e776a3dfb0197c4@mail.gmail.com>
	 <4B0EA11E.8090905@Gmail.com>
	 <8211a1320911260741i68cce53dib0d04a8bf802a835@mail.gmail.com>
	 <4B0EA5C0.9000908@Gmail.com>
	 <8211a1320911260805h747a3f06j33946a4a40d7c04@mail.gmail.com>
 <314098690911261014m2c77d395oc1eb243a42d7e611@mail.gmail.com>
In-Reply-To: <314098690911261014m2c77d395oc1eb243a42d7e611@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------050602020407080906050109"

--------------050602020407080906050109
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


> Are the record processing steps bound by a local machine resource - cpu,
> disk io or other?
>   
Some disk I/O. Not so much compared with the CPU. Basically it is a CPU 
bound. This is why each machine has 16 cores.
> What I often do when I have lots of small files to handle is use the
> NlineInputFormat,
Each file contains a complete/independent set of records. I cannot mix 
the data resulted from processing two different files.


---------
Ok. I think I need to re-explain my problem :)
While running jobs on these small files, the computation time was almost 
5 times longer than expected. It looks like the job was affected by the 
number of map task that I have (100). I don't know which are the best 
parameters in my case (10MB files).

I have zero reduce tasks.


--------------050602020407080906050109--