Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com
 designates 209.85.161.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=miAhAwMafr/qSTYQptkWI2mbyxWXTzMCccL7Z1f+mQi7EXuIKPsPaz3BJPBv7FuBHe
         W2CTeCf7qcdo+Lrs0PLGE673XGHNM5c56pAde+2Fq5LArJ2+Nn+LPWNP3ODs6lgvDrCP
         Mb0WNS7ukUwEkcori8f6cOhtZKIPJyA5JTTjU=
MIME-Version: 1.0
In-Reply-To: <1291073176029-1989598.post@n3.nabble.com>
References: <1291073176029-1989598.post@n3.nabble.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Tue, 30 Nov 2010 13:51:31 +0530
Message-ID: <AANLkTinz3fbzXwq0Vo_z=Wq4iTq7CaO_9u2UK6UpE-iy@mail.gmail.com>
Subject: Re: small files and number of mappers
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hey,

On Tue, Nov 30, 2010 at 4:56 AM, Marc Sturlese <marc.sturlese@gmail.com> wrote:
>
> Hey there,
> I am doing some tests and wandering which are the best practices to deal
> with very small files which are continuously being generated(1Mb or even
> less).

Have a read: http://www.cloudera.com/blog/2009/02/the-small-files-problem/

>
> I see that if I have hundreds of small files in hdfs, hadoop automatically
> will create A LOT of map tasks to consume them. Each map task will take 10
> seconds or less... I don't know if it's possible to change the number of map
> tasks from java code using the new API (I know it can be done with the old
> one). I would like to do something like NumMapTasksCalculatedByHadoop * 0.3.
> This way, less maps tasks would be instanciated and each would be working
> more time.

Perhaps you need to use MultiFileInputFormat:
http://www.cloudera.com/blog/2009/02/the-small-files-problem/

-- 
Harsh J
www.harshj.com