Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id AAC0D200C14 for ; Tue, 7 Feb 2017 17:55:36 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id A92B2160B3E; Tue, 7 Feb 2017 16:55:36 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A750F160B32 for ; Tue, 7 Feb 2017 17:55:35 +0100 (CET) Received: (qmail 95765 invoked by uid 500); 7 Feb 2017 16:55:34 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 95752 invoked by uid 99); 7 Feb 2017 16:55:34 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Feb 2017 16:55:34 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 4F1501A0490 for ; Tue, 7 Feb 2017 16:55:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.798 X-Spam-Level: * X-Spam-Status: No, score=1.798 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 5P9Wou73FiiY for ; Tue, 7 Feb 2017 16:55:30 +0000 (UTC) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 7D12D5F570 for ; Tue, 7 Feb 2017 16:55:30 +0000 (UTC) Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.20/8.16.0.20) with SMTP id v17Gi7Z6035969 for ; Tue, 7 Feb 2017 11:55:08 -0500 Received: from e06smtp08.uk.ibm.com (e06smtp08.uk.ibm.com [195.75.94.104]) by mx0b-001b2d01.pphosted.com with ESMTP id 28fch527pf-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Tue, 07 Feb 2017 11:55:08 -0500 Received: from localhost by e06smtp08.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 7 Feb 2017 16:55:05 -0000 Received: from d06dlp03.portsmouth.uk.ibm.com (9.149.20.15) by e06smtp08.uk.ibm.com (192.168.101.138) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 7 Feb 2017 16:55:04 -0000 X-IBM-Helo: d06dlp03.portsmouth.uk.ibm.com X-IBM-MailFrom: MASSIMIL@il.ibm.com X-IBM-RcptTo: user@accumulo.apache.org Received: from b06cxnps4076.portsmouth.uk.ibm.com (d06relay13.portsmouth.uk.ibm.com [9.149.109.198]) by d06dlp03.portsmouth.uk.ibm.com (Postfix) with ESMTP id B43FA1B08061 for ; Tue, 7 Feb 2017 16:57:56 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id v17Gt3QB59310102 for ; Tue, 7 Feb 2017 16:55:03 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 586B54C040 for ; Tue, 7 Feb 2017 16:54:56 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CBDA44C04A for ; Tue, 7 Feb 2017 16:54:55 +0000 (GMT) Received: from d50lp32.co.us.ibm.com (unknown [9.17.249.36]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTPS for ; Tue, 7 Feb 2017 16:54:55 +0000 (GMT) Received: from localhost by d50lp32.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 7 Feb 2017 09:55:01 -0700 Received: from smtp.notes.na.collabserv.com (192.155.248.73) by d50lp32.co.us.ibm.com (192.168.2.143) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128/128) Tue, 7 Feb 2017 09:54:59 -0700 Received: from localhost by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Tue, 7 Feb 2017 16:54:57 -0000 Received: from us1a3-smtp06.a3.dal06.isc4sb.com (10.146.103.243) by smtp.notes.na.collabserv.com (10.106.227.90) with smtp.notes.na.collabserv.com ESMTP; Tue, 7 Feb 2017 16:54:54 -0000 Received: from us1a3-mail112.a3.dal06.isc4sb.com ([10.146.45.94]) by us1a3-smtp06.a3.dal06.isc4sb.com with ESMTP id 2017020716545366-392820 ; Tue, 7 Feb 2017 16:54:53 +0000 In-Reply-To: To: user@accumulo.apache.org Subject: Re: how to make LZO compression work? From: "Massimilian Mattetti" Date: Tue, 7 Feb 2017 18:54:54 +0200 References: MIME-Version: 1.0 X-KeepSent: B0DFA7E3:6F71E80C-C22580C0:005C5F81; type=4; name=$KeepSent X-Mailer: IBM Notes Release 9.0.1EXT SHF766 December 14, 2016 X-LLNOutbound: False X-Disclaimed: 51207 X-TNEFEvaluated: 1 Content-Type: multipart/alternative; boundary="=_alternative 005CE9DDC22580C0_=" x-cbid: 17020716-0032-0000-0000-00000718CDEB X-IBM-ISS-SpamDetectors: Score=0.4332; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.4332; ST=0; TS=0; UL=0; ISC= X-IBM-ISS-DetailInfo: BY=3.00006573; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000202; SDB=6.00818541; UDB=6.00400009; UTC=2017-02-07 16:54:55 x-cbparentid: 17020716-3108-0000-0000-000002E1344C X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused X-TM-AS-GCONF: 00 X-Content-Scanned: Fidelis XPS MAILER X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-02-07_09:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=1 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1612050000 definitions=main-1702070160 archived-at: Tue, 07 Feb 2017 16:55:36 -0000 --=_alternative 005CE9DDC22580C0_= Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="US-ASCII" I did the two steps you suggested and it worked. Thanks.=20 Regards, Max From: Sean Busbey To: Accumulo User List Date: 07/02/2017 18:00 Subject: Re: how to make LZO compression work? The documentation put out by Cloudera for running Accumulo on CDH used to have a section on getting LZO working[1]. One thing I see in those docs that I don't see you doing an equivalent of is setting LD=5FLIBRARY=5FPATH to point to the LZO libraries. They also specify io.compression.codec.lzo.class as a command line argument for the service startup rather than putting it in core-site.xml, but it's not clear to me if that's necessary. [1]:=20 http://www.cloudera.com/documentation/other/accumulo/1-6-0/PDF/Apache-Accum= ulo-Installation-Guide.pdf On Tue, Feb 7, 2017 at 5:18 AM, Massimilian Mattetti wrote: > Hi all, > > I got stuck trying to enable the LZO compression on a table. I installed = the > native-lzo library on each tablet server (sudo apt-get install=20 liblzo2-dev) > and set the compression codecs libraries in the Hadoop core-site.xml: > > io.compression.codecs > org.apache.hadoop.io.compress.GzipCodec, > org.apache.hadoop.io.compress.DefaultCodec, > org.apache.hadoop.io.compress.BZip2Codec, > com.hadoop.compression.lzo.LzoCodec, > com.hadoop.compression.lzo.LzopCodec > > > > io.compression.codec.lzo.class > com.hadoop.compression.lzo.LzoCodec > > > I also added the path to the library to the general.classpaths property = of > accumulo-site.xml: > > general.classpaths > > .... > > /usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.5.0.0-1245.jar, > >=20 /usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64/libgplcompression.= so, > /lib/x86=5F64-linux-gnu/liblzo2.so.2 > > > > I tested that the lzo library is working correctly compressing a file=20 via a > MapReduce Job, but when I asked Accumulo to compact the table on which I > enabled the LZO compression I got this error on each tablet server: > MajC Unexpected exception, extent =3D 1z; ? ??????????<; ? ??????????. > java.lang.RuntimeException: native-lzo library not available > at > com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155) > at >=20 org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150) > at >=20 org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165) > at >=20 org.apache.accumulo.core.file.rfile.bcfile.Compression$Algorithm.getCompres= sor(Compression.java:502) > at >=20 org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer$WBlockState.= (BCFile.java:192) > at >=20 org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer.prepareDataBlock(B= CFile.java:487) > at >=20 org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Writer.prepa= reDataBlock(CachableBlockFile.java:89) > at >=20 org.apache.accumulo.core.file.rfile.RFile$LocalityGroupWriter.append(RFile.= java:429) > at > org.apache.accumulo.core.file.rfile.RFile$Writer.append(RFile.java:596) > at >=20 org.apache.accumulo.core.file.BloomFilterLayer$Writer.append(BloomFilterLay= er.java:141) > at >=20 org.apache.accumulo.tserver.tablet.Compactor.compactLocalityGroup(Compactor= .java:367) > at > org.apache.accumulo.tserver.tablet.Compactor.call(Compactor.java:223) > at >=20 org.apache.accumulo.tserver.tablet.Tablet.=5FmajorCompact(Tablet.java:1850) > at > org.apache.accumulo.tserver.tablet.Tablet.majorCompact(Tablet.java:1967) > at >=20 org.apache.accumulo.tserver.tablet.CompactionRunner.run(CompactionRunner.ja= va:44) > at > org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57) > at >=20 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1= 142) > at >=20 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:= 617) > at >=20 org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35) > at java.lang.Thread.run(Thread.java:745) > What am I missing? > Thanks > > Regards, > Max > --=20 busbey --=_alternative 005CE9DDC22580C0_= Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="US-ASCII" I did the two steps you suggested and it worked.
Thanks.
Regards,
Max





From: &= nbsp;      Sean Busbey <busbey@clou= dera.com>
T= o:        Accumulo User List <user@accumulo.apache.org>
Date:        07/02/2017 18:00
= Subject:        Re: how to make LZO compression work?



= The documentation put out by Cloudera for running Accumulo on CDH used
to have a section on getting LZO working[1].
One thing I see in those docs that I don't see you doing an equivalent
= of is setting LD=5FLIBRARY=5FPATH to point to the LZO libraries. They also<= br>specify io.compression.codec.lzo.class as a command line argument forthe service startup rather than putting it in core-site.xml, but it's
n= ot clear to me if that's necessary.

[1]: http://www.cloudera.com/document= ation/other/accumulo/1-6-0/PDF/Apache-Accumulo-Installation-Guide.pdf

On Tue, Feb 7, 2017 at 5:18 AM, Massim= ilian Mattetti
<MASSIMIL@il.ibm.com> wrote:
> Hi all,
>= ;
> I got stuck trying to enable the LZO compression on a table. I in= stalled the
> native-lzo library on each tablet server (sudo apt-get install = liblzo2-dev)
> and set the compression codecs libraries in the Hadoop= core-site.xml:
> <property>
>   <name>io.comp= ression.codecs</name>
>   <value>org.apache.hadoop.i= o.compress.GzipCodec,
>       org.apache.hadoop.io.com= press.DefaultCodec,
>       org.apache.hadoop.io.compr= ess.BZip2Codec,
>       com.hadoop.compression.lzo.Lzo= Codec,
>       com.hadoop.compression.lzo.LzopCodec>   </value>
> </property>
> <property&= gt;
>   <name>io.compression.codec.lzo.class</name><= br>>   <value>com.hadoop.compression.lzo.LzoCodec</value&g= t;
> </property>
>
>  I also added the path to= the library to the general.classpaths property of
> accumulo-site.xml:
> <property>
> &nb= sp;     <name>general.classpaths</name>
>  = ;     <value>
>           &n= bsp;   ....
>
> /usr/hdp/current/hadoop-client/lib/hadoop-= lzo-0.6.0.2.5.0.0-1245.jar,
>
> /usr/hdp/current/hadoop-client/= lib/native/Linux-amd64-64/libgplcompression.so,
>      = ;           /lib/x86=5F64-linux-gnu/liblzo2.so.2>         </value>
> </property>=
>
> I tested that the lzo library is working correctly compres= sing a file via a
> MapReduce Job, but when I asked Accumulo to compact the table= on which I
> enabled the LZO compression I got this error on each tablet serve= r:
> MajC Unexpected exception, extent =3D 1z; ? ??????????<; ? ??= ????????.
>         java.lang.RuntimeException: n= ative-lzo library not available
>             &nb= sp;   at
> com.hadoop.compression.lzo.LzoCodec.getCompressorType= (LzoCodec.java:155)
>              = ;   at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(C= odecPool.java:150)
>              =   at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(Co= decPool.java:165)
>               =   at
> org.apache.accumulo.core.file.rfile.bcfile.Compression$Al= gorithm.getCompressor(Compression.java:502)
>       &n= bsp;         at
> org.apache.accumulo.core.file.r= file.bcfile.BCFile$Writer$WBlockState.<init>(BCFile.java:192)
>=                 at
> org.apa= che.accumulo.core.file.rfile.bcfile.BCFile$Writer.prepareDataBlock(BCFile.j= ava:487)
>                 at=
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Wri= ter.prepareDataBlock(CachableBlockFile.java:89)
>      = ;           at
> org.apache.accumulo.core.fi= le.rfile.RFile$LocalityGroupWriter.append(RFile.java:429)
>   &n= bsp;             at
> org.apache.accumu= lo.core.file.rfile.RFile$Writer.append(RFile.java:596)
>    = ;             at
> org.apache.accumulo.= core.file.BloomFilterLayer$Writer.append(BloomFilterLayer.java:141)
>=                 at
> org.apa= che.accumulo.tserver.tablet.Compactor.compactLocalityGroup(Compactor.java:3= 67)
>                 at
&= gt; org.apache.accumulo.tserver.tablet.Compactor.call(Compactor.java:223)>                 at
> o= rg.apache.accumulo.tserver.tablet.Tablet.=5FmajorCompact(Tablet.java:1850)<= br>>                 at
> = org.apache.accumulo.tserver.tablet.Tablet.majorCompact(Tablet.java:1967)>                 at
> or= g.apache.accumulo.tserver.tablet.CompactionRunner.run(CompactionRunner.java= :44)
>                 at
= > org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)>                 at
> ja= va.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:114= 2)
>                 at
&g= t; java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= va:617)
>                 at<= br>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.j= ava:35)
>                 at = java.lang.Thread.run(Thread.java:745)
> What am I missing?
> Th= anks
>
> Regards,
> Max
>



--
bus= bey




--=_alternative 005CE9DDC22580C0_=--