Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of mailinglists19@gmail.com
 designates 209.85.214.43 as permitted sender)
MIME-Version: 1.0
Date: Mon, 11 Feb 2013 10:22:05 -0800
Message-ID: 
 <CAHXz3_Fmrt59_2DKuZyhNgHW3BQFsdqesLw4+Fx03EdWadnQeg@mail.gmail.com>
Subject: Loader for small files
From: Something Something <mailinglists19@gmail.com>
To: user@pig.apache.org, hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175885587b4c6304d576fce5

--0015175885587b4c6304d576fce5
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hello,

We are running into performance issues with Pig/Hadoop because our input
files are small.  Everything goes to only 1 Mapper.  To get around this, we
are trying to use our own Loader like this:

1)  Extend PigStorage:

public class SmallFileStorage extends PigStorage {

    public SmallFileStorage(String delimiter) {
        super(delimiter);
    }

    @Override
    public InputFormat getInputFormat() {
        return new NLineInputFormat();
    }
}


2)  Add command line argument to the Pig command as follows:

-Dmapreduce.input.lineinputformat.linespermap=3D500000


3)  Use SmallFileStorage in the Pig script as follows:

USING com.xxx.yyy.SmallFileStorage ('\t')


But this doesn't seem to work.  We still see that everything is going to
one mapper.  Before we spend any more time on this, I am wondering if this
is a good approach =96 OR =96 if there's a better approach?  Please let me
know.  Thanks.

--0015175885587b4c6304d576fce5--