Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of kuosenhao@gmail.com designates
 209.85.216.201 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=t2ZZmbGdIM9nwm6QkA0NO1M3tyu9xU6JXlCDiHyweQV4w5npgWaLmeinggW2nAiMPv
         eaT7lLCTwdTRmmeO1/sxigRj2meei9Lq3M41rAewsieRR6NZ9eZ7Q4Vxvv3WyVxO7caG
         goSRHg2Cba1eN4VZXiZa6A8HEFc7aigCJEDpg=
MIME-Version: 1.0
In-Reply-To: <b3d92c700912310953u792c3e33o6d976102ffcd50a1@mail.gmail.com>
References: <b3d92c700912310953u792c3e33o6d976102ffcd50a1@mail.gmail.com>
Date: Thu, 31 Dec 2009 12:21:06 -0800
Message-ID: <b3d92c700912311221s6697a58ydc537db584033409@mail.gmail.com>
Subject: Re: How to ensure LzoTextInputFormat is used to generate input splits
	for .lzo files
From: Steve Kuo <kuosenhao@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502c10daf0536047c0bfeb4

--00504502c10daf0536047c0bfeb4
Content-Type: text/plain; charset=ISO-8859-1

Digging around the new Job api with a rested brain came up with

             job.setInputFormatClass(LzoTextInputFormat.class);

that solved the problem.

On Thu, Dec 31, 2009 at 9:53 AM, Steve Kuo <kuosenhao@gmail.com> wrote:

> I have followed
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/and
> http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ to build the
> requisite hadoop-lzo jar and native .so files.  (The jar and .so files were
> built from Kevin Weil's git repository.  Thanks Kevin.)  I have configured
> core-site.xml and mapred-site.xml as instructed to enable lzo for both map
> and reduce output.  The creation of lzo index also worked. The last step was
> to replace TextInputFormat with LzoTextInputFormat.  As I only have
>
>     FileInputFormat.addInputPath(jobConf, new Path(inputPath));
>
> it was replaced with
>
>      LzoTextInputFormat.addInputPath(job, new Path(inputPath));
>
> When I ran my MR job, I noticed that the new code was able to read in .lzo
> input files and decompressed fine.   The output was also lzo compressed.
> However, only one map job was created for each input .lzo file indicating
> that input splitting was not done by LzoTextInputFormat but more likely by
> its parent such as FileInputFormat.  There must be a way to ensure
> LzoTextInputFormat is used in the Map task.  How can this be done?
>
> Thanks in advance.
>
>

--00504502c10daf0536047c0bfeb4--