Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of billmcn@gmail.com designates
 209.85.220.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:cc:content-type;
        b=KOlvP3AxEGccwg5v7i8V8aUV2USEKc3amW2RROyaLJkDLcILQGW4sN5JFjihC1nR+z
         4vjhPx/b/z1JQQoJF7TXNZxMHTm8P3IEraGbSxjUuWcrQY5ChxCIVnFDKpB+/SEc8lLV
         9PmaHwEcUm6sJjZsWN3jBJaR6kaqxvznF2EXw=
MIME-Version: 1.0
Date: Tue, 29 Mar 2011 11:18:17 -0700
Message-ID: <AANLkTik7TOStvyPhG0jNLYuA7NRZs+kOCupyto_zXmxC@mail.gmail.com>
Subject: How do I increase mapper granularity?
From: "W.P. McNeill" <billmcn@gmail.com>
To: common-user@hadoop.apache.org
Cc: Siddharth Agrawal <sagrawal@intelius.com>,
 Andrew Borthwick <aborthwick@intelius.com>
Content-Type: multipart/alternative; boundary=0016e646a4d09d8558049fa315eb

--0016e646a4d09d8558049fa315eb
Content-Type: text/plain; charset=UTF-8

I'm running a job whose mappers take a long time, which causes problems like
starving out other jobs that want to run on the same cluster.  Rewriting the
mapper algorithm is not currently an option, but I still need a way to
increase the number of mappers so that I will have greater granularity.
 What is the best way to do this?

Looking through the O'Reilly book and starting from
this<http://wiki.apache.org/hadoop/HowManyMapsAndReduces>Wiki page
I've come up with a couple of ideas:

   1. Set mapred.map.tasks to the value I want.
   2. Decrease the block size of my input files.

What are the gotchas with these approaches?  I know that (1) may not work
because this parameter is just a suggestion.  Is there a command line option
that accomplishes (2), or do I have to do a distcp with a non-default block
size.  (I think the answer is that I have to do a distcp, but I'm making
sure.)

Are there other approaches?  Are there other gotchas that come with trying
to increase mapper granularity.  I know this can be more of an art than a
science.

Thanks.

--0016e646a4d09d8558049fa315eb--