Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: pass (athena.apache.org: domain of vipandey@gmail.com designates
 209.85.192.176 as permitted sender)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\))
Subject: Re: LZO support in Spark 1.0.0 - nothing seems to work
From: Vipul Pandey <vipandey@gmail.com>
In-Reply-To: <1411000826195-14494.post@n3.nabble.com>
Date: Wed, 17 Sep 2014 17:51:42 -0700
Cc: user@spark.incubator.apache.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <9F298CDE-9E28-4AA8-BC8B-D13D853ED833@gmail.com>
References: <1411000826195-14494.post@n3.nabble.com>
To: rogthefrog <roger@amino.com>

It works for me :=20


export =
JAVA_LIBRARY_PATH=3D$JAVA_LIBRARY_PATH:/opt/cloudera/parcels/HADOOP_LZO/li=
b/hadoop/lib/native
export =
LD_LIBRARY_PATH=3D$LD_LIBRARY_PATH:/opt/cloudera/parcels/HADOOP_LZO/lib/ha=
doop/lib/native

export =
SPARK_LIBRARY_PATH=3D$SPARK_LIBRARY_PATH:/opt/cloudera/parcels/HADOOP_LZO/=
lib/hadoop/lib/native
export =
SPARK_CLASSPATH=3D$SPARK_CLASSPATH:/opt/cloudera/parcels/HADOOP_LZO/lib/ha=
doop/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar


I hope you are adding this to the code :=20

    val conf =3D sc.hadoopConfiguration
    =
conf.set("io.compression.codecs","com.hadoop.compression.lzo.LzopCodec")


Vipul

On Sep 17, 2014, at 5:40 PM, rogthefrog <roger@amino.com> wrote:

> I have a HDFS cluster managed with CDH Manager. Version is CDH 5.1 =
with
> matching GPLEXTRAS parcel. LZO works with Hive and Pig, but I can't =
make it
> work with Spark 1.0.0. I've tried:
>=20
> * Setting this:
>=20
> HADOOP_OPTS=3D"-Djava.net.preferIPv4Stack=3Dtrue $HADOOP_CLIENT_OPTS
> =
-Djava.library.path=3D/opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/nativ=
e/"
>=20
> * Setting this in spark-env.sh. I tried with and without "export". I =
tried
> in CDH Manager and manually on the host.
>=20
> export
> =
SPARK_CLASSPATH=3D$SPARK_CLASSPATH:/opt/cloudera/parcels/GPLEXTRAS/lib/had=
oop/lib/hadoop-lzo.jar
> export
> =
SPARK_LIBRARY_PATH=3D$SPARK_LIBRARY_PATH:/opt/cloudera/parcels/GPLEXTRAS/l=
ib/hadoop/lib/native/
>=20
> * Setting this in /etc/spark/conf/spark-defaults.conf:
>=20
> spark.executor.extraLibraryPath=20
> /opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/native
> spark.spark.executor.extraClassPath
> /opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/hadoop-lzo.jar
>=20
> * Adding this in CDH manager:
>=20
> export =
LD_LIBRARY_PATH=3D/opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/native
>=20
> * Hardcoding
> =
-Djava.library.path=3D/opt/cloudera/parcels/GPLEXTRAS/lib/hadoop/lib/nativ=
e in
> the Spark command=20
>=20
> * Symlinking the gpl compression binaries into
> /opt/cloudera/parcels/CDH/lib/hadoop/lib/native
>=20
> * Symlinking the gpl compression binaries into /usr/lib
>=20
> And nothing worked. When I run pyspark I get this:
>=20
> 14/09/17 20:38:54 WARN util.NativeCodeLoader: Unable to load =
native-hadoop
> library for your platform... using builtin-java classes where =
applicable
>=20
> and when I try to run a simple job on a LZO file in HDFS I get this:
>=20
> distFile.count()
> 14/09/17 13:51:54 ERROR GPLNativeCodeLoader: Could not load native gpl
> library
> java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
>       at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
>       at java.lang.Runtime.loadLibrary0(Runtime.java:849)
>       at java.lang.System.loadLibrary(System.java:1088)
>       at
> =
com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoade=
r.java:32)
>       at =
com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:71)
>=20
> Can anybody help please? Many thanks.
>=20
>=20
>=20
> --
> View this message in context: =
http://apache-spark-user-list.1001560.n3.nabble.com/LZO-support-in-Spark-1=
-0-0-nothing-seems-to-work-tp14494.html
> Sent from the Apache Spark User List mailing list archive at =
Nabble.com.
>=20
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>=20


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org