hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: LZO with sequenceFile
Date Sun, 26 Feb 2012 17:09:34 GMT
If you want to just quickly package the hadoop-lzo items instead of
building/managing-deployment on your own, you can reuse Todd Lipcon's
script at https://github.com/toddlipcon/hadoop-lzo-packager - Creates
both RPMs and DEBs.

On Sun, Feb 26, 2012 at 9:55 PM, Ioan Eugen Stan <stan.ieugen@gmail.com> wrote:
> 2012/2/26 Mohit Anchlia <mohitanchlia@gmail.com>:
>> Thanks. Does it mean LZO is not installed by default? How can I install LZO?
>
> The LZO library is released under GPL and I believe it can't be
> included in most distributions of Hadoop because of this (can't mix
> GPL with non GPL stuff). It should be easily available though.
>
>> On Sat, Feb 25, 2012 at 6:27 PM, Shi Yu <shiyu@uchicago.edu> wrote:
>>
>>> Yes, it is supported by Hadoop sequence file. It is splittable
>>> by default. If you have installed and specified LZO correctly,
>>> use these:
>>>
>>>
>>> org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputForma
>>> t.setCompressOutput(job,true);
>>>
>>> org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputForma
>>> t.setOutputCompressorClass(job,com.hadoop.compression.lzo.LzoC
>>> odec.class);
>>>
>>> org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputForma
>>> t.setOutputCompressionType(job,
>>> SequenceFile.CompressionType.BLOCK);
>>>
>>> job.setOutputFormatClass(org.apache.hadoop.mapreduce.lib.outpu
>>> t.SequenceFileOutputFormat.class);
>>>
>>>
>>> Shi
>>>
>
>
>
> --
> Ioan Eugen Stan
> http://ieugen.blogspot.com/



-- 
Harsh J

Mime
View raw message