spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ruslan Dautkhanov <dautkha...@gmail.com>
Subject Re: Problem after enabling Hadoop native libraries
Date Tue, 30 Jun 2015 20:55:12 GMT
You can run
hadoop checknative -a
and see if bzip2 is detected correctly.


-- 
Ruslan Dautkhanov

On Fri, Jun 26, 2015 at 10:18 AM, Marcelo Vanzin <vanzin@cloudera.com>
wrote:

> What master are you using? If this is not a "local" master, you'll need to
> set LD_LIBRARY_PATH on the executors also (using
> spark.executor.extraLibraryPath).
>
> If you are using local, then I don't know what's going on.
>
> On Fri, Jun 26, 2015 at 1:39 AM, Arunabha Ghosh <arunabha.gh@gmail.com>
> wrote:
>
>> Hi,
>>      I'm having trouble reading Bzip2 compressed sequence files after I
>> enabled hadoop native libraries in spark.
>>
>> Running
>> LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/ $SPARK_HOME/bin/spark-submit
>> --class .... gives the following error
>>
>> 5/06/26 00:48:02 INFO CodecPool: Got brand-new decompressor [.bz2]
>> 15/06/26 00:48:02 ERROR Executor: Exception in task 3.0 in stage 0.0 (TID
>> 3)
>> java.lang.UnsupportedOperationException
>> at
>> org.apache.hadoop.io.compress.bzip2.BZip2DummyDecompressor.decompress(BZip2DummyDecompressor.java:32)
>> at
>> org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91)
>> at
>> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
>> at java.io.DataInputStream.readFully(DataInputStream.java:195)
>> at java.io.DataInputStream.readLong(DataInputStream.java:416)
>>
>> removing the LD_LIBRARY_PATH makes spark run fine but it gives the
>> following warning
>> WARN NativeCodeLoader: Unable to load native-hadoop library for your
>> platform... using builtin-java classes where applicable
>>
>> Has anyone else run into this issue ? Any help is welcome.
>>
>
>
>
> --
> Marcelo
>

Mime
View raw message