accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Massimilian Mattetti" <MASSI...@il.ibm.com>
Subject Re: how to make LZO compression work?
Date Tue, 07 Feb 2017 16:54:54 GMT
I did the two steps you suggested and it worked.
Thanks. 

Regards,
Max





From:   Sean Busbey <busbey@cloudera.com>
To:     Accumulo User List <user@accumulo.apache.org>
Date:   07/02/2017 18:00
Subject:        Re: how to make LZO compression work?



The documentation put out by Cloudera for running Accumulo on CDH used
to have a section on getting LZO working[1].

One thing I see in those docs that I don't see you doing an equivalent
of is setting LD_LIBRARY_PATH to point to the LZO libraries. They also
specify io.compression.codec.lzo.class as a command line argument for
the service startup rather than putting it in core-site.xml, but it's
not clear to me if that's necessary.

[1]: 
http://www.cloudera.com/documentation/other/accumulo/1-6-0/PDF/Apache-Accumulo-Installation-Guide.pdf


On Tue, Feb 7, 2017 at 5:18 AM, Massimilian Mattetti
<MASSIMIL@il.ibm.com> wrote:
> Hi all,
>
> I got stuck trying to enable the LZO compression on a table. I installed 
the
> native-lzo library on each tablet server (sudo apt-get install 
liblzo2-dev)
> and set the compression codecs libraries in the Hadoop core-site.xml:
> <property>
>   <name>io.compression.codecs</name>
>   <value>org.apache.hadoop.io.compress.GzipCodec,
>       org.apache.hadoop.io.compress.DefaultCodec,
>       org.apache.hadoop.io.compress.BZip2Codec,
>       com.hadoop.compression.lzo.LzoCodec,
>       com.hadoop.compression.lzo.LzopCodec
>   </value>
> </property>
> <property>
>   <name>io.compression.codec.lzo.class</name>
>   <value>com.hadoop.compression.lzo.LzoCodec</value>
> </property>
>
>  I also added the path to the library to the general.classpaths property 
of
> accumulo-site.xml:
> <property>
>       <name>general.classpaths</name>
>       <value>
>               ....
>
> /usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.5.0.0-1245.jar,
>
> 
/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64/libgplcompression.so,
>                 /lib/x86_64-linux-gnu/liblzo2.so.2
>         </value>
> </property>
>
> I tested that the lzo library is working correctly compressing a file 
via a
> MapReduce Job, but when I asked Accumulo to compact the table on which I
> enabled the LZO compression I got this error on each tablet server:
> MajC Unexpected exception, extent = 1z; ? ??????????<; ? ??????????.
>         java.lang.RuntimeException: native-lzo library not available
>                 at
> com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
>                 at
> 
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
>                 at
> 
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
>                 at
> 
org.apache.accumulo.core.file.rfile.bcfile.Compression$Algorithm.getCompressor(Compression.java:502)
>                 at
> 
org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer$WBlockState.<init>(BCFile.java:192)
>                 at
> 
org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer.prepareDataBlock(BCFile.java:487)
>                 at
> 
org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Writer.prepareDataBlock(CachableBlockFile.java:89)
>                 at
> 
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupWriter.append(RFile.java:429)
>                 at
> org.apache.accumulo.core.file.rfile.RFile$Writer.append(RFile.java:596)
>                 at
> 
org.apache.accumulo.core.file.BloomFilterLayer$Writer.append(BloomFilterLayer.java:141)
>                 at
> 
org.apache.accumulo.tserver.tablet.Compactor.compactLocalityGroup(Compactor.java:367)
>                 at
> org.apache.accumulo.tserver.tablet.Compactor.call(Compactor.java:223)
>                 at
> 
org.apache.accumulo.tserver.tablet.Tablet._majorCompact(Tablet.java:1850)
>                 at
> org.apache.accumulo.tserver.tablet.Tablet.majorCompact(Tablet.java:1967)
>                 at
> 
org.apache.accumulo.tserver.tablet.CompactionRunner.run(CompactionRunner.java:44)
>                 at
> org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
>                 at
> 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>                 at
> 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>                 at
> 
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>                 at java.lang.Thread.run(Thread.java:745)
> What am I missing?
> Thanks
>
> Regards,
> Max
>



-- 
busbey






Mime
View raw message