Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))
Subject: Re: Loader for small files
From: David LaBarbera <davidlabarbera@localresponse.com>
In-Reply-To: 
 <CAHXz3_EKcHaynsLXfELnQ76z8yESL_VPL0na8jexcz5seQLsMQ@mail.gmail.com>
Date: Mon, 11 Feb 2013 13:29:56 -0500
Cc: user@hadoop.apache.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <4300198D-0743-4D51-9840-5130FA9D5E7A@localresponse.com>
References: 
 <CAHXz3_Fmrt59_2DKuZyhNgHW3BQFsdqesLw4+Fx03EdWadnQeg@mail.gmail.com>
 <CAHXz3_EKcHaynsLXfELnQ76z8yESL_VPL0na8jexcz5seQLsMQ@mail.gmail.com>
To: user@pig.apache.org

You could store your data in smaller block sizes. Do something like
hadoop fs HADOOP_OPTS=3D"-Ddfs.block.size=3D1048576 =
-Dfs.local.block.size=3D1048576" -cp /org-input /small-block-input
You might only need one of those parameters. You can verify the block =
size with
hadoop fsck /small-block-input

In your pig script, you'll probably need to set
pig.maxCombinedSplitSize=20
to something around the block size

David

On Feb 11, 2013, at 1:24 PM, Something Something =
<mailinglists19@gmail.com> wrote:

> Sorry.. Moving 'hbase' mailing list to BCC 'cause this is not related =
to
> HBase.  Adding 'hadoop' user group.
>=20
> On Mon, Feb 11, 2013 at 10:22 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>=20
>> Hello,
>>=20
>> We are running into performance issues with Pig/Hadoop because our =
input
>> files are small.  Everything goes to only 1 Mapper.  To get around =
this, we
>> are trying to use our own Loader like this:
>>=20
>> 1)  Extend PigStorage:
>>=20
>> public class SmallFileStorage extends PigStorage {
>>=20
>>    public SmallFileStorage(String delimiter) {
>>        super(delimiter);
>>    }
>>=20
>>    @Override
>>    public InputFormat getInputFormat() {
>>        return new NLineInputFormat();
>>    }
>> }
>>=20
>>=20
>>=20
>> 2)  Add command line argument to the Pig command as follows:
>>=20
>> -Dmapreduce.input.lineinputformat.linespermap=3D500000
>>=20
>>=20
>>=20
>> 3)  Use SmallFileStorage in the Pig script as follows:
>>=20
>> USING com.xxx.yyy.SmallFileStorage ('\t')
>>=20
>>=20
>> But this doesn't seem to work.  We still see that everything is going =
to
>> one mapper.  Before we spend any more time on this, I am wondering if =
this
>> is a good approach =96 OR =96 if there's a better approach?  Please =
let me
>> know.  Thanks.
>>=20
>>=20
>>=20