hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: How to ensure LzoTextInputFormat is used to generate input splits for .lzo files
Date Mon, 18 Jan 2010 19:12:52 GMT
For our custom text-based file format, we use empty line to mark data for
different households.
Can we make LZO block start to be aligned with new household, possibly by
modifying LzoIndexRecordWriter ?

Thanks

On Thu, Dec 31, 2009 at 3:44 PM, Kevin Weil <kevinweil@gmail.com> wrote:

> Steve, glad you got it figured out.  Interested to hear how it goes, and of
> course feel free to post bugs/requests to the github page
> www.github.com/kevinweil/hadoop-lzo.
>
> Kevin
>
> On Thu, Dec 31, 2009 at 12:21 PM, Steve Kuo <kuosenhao@gmail.com> wrote:
>
> > Digging around the new Job api with a rested brain came up with
> >
> >             job.setInputFormatClass(LzoTextInputFormat.class);
> >
> > that solved the problem.
> >
> > On Thu, Dec 31, 2009 at 9:53 AM, Steve Kuo <kuosenhao@gmail.com> wrote:
> >
> > > I have followed
> > >
> >
> http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/and
> > > http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ to build the
> > > requisite hadoop-lzo jar and native .so files.  (The jar and .so files
> > were
> > > built from Kevin Weil's git repository.  Thanks Kevin.)  I have
> > configured
> > > core-site.xml and mapred-site.xml as instructed to enable lzo for both
> > map
> > > and reduce output.  The creation of lzo index also worked. The last
> step
> > was
> > > to replace TextInputFormat with LzoTextInputFormat.  As I only have
> > >
> > >     FileInputFormat.addInputPath(jobConf, new Path(inputPath));
> > >
> > > it was replaced with
> > >
> > >      LzoTextInputFormat.addInputPath(job, new Path(inputPath));
> > >
> > > When I ran my MR job, I noticed that the new code was able to read in
> > .lzo
> > > input files and decompressed fine.   The output was also lzo
> compressed.
> > > However, only one map job was created for each input .lzo file
> indicating
> > > that input splitting was not done by LzoTextInputFormat but more likely
> > by
> > > its parent such as FileInputFormat.  There must be a way to ensure
> > > LzoTextInputFormat is used in the Map task.  How can this be done?
> > >
> > > Thanks in advance.
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message