hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kiru Pakkirisamy <kirupakkiris...@yahoo.com>
Subject Re: lzo error while running mr job
Date Tue, 27 Oct 2015 15:19:41 GMT
Harish,

We don't have lzo in the io.compression.codecs list.

That is what is puzzling me.

Regards, 

Kiru 

From:"Harsh J" <harsh@cloudera.com>
Date:Mon, Oct 26, 2015 at 11:39 PM
Subject:Re: lzo error while running mr job

Every codec in the io.compression.codecs list of classes will be initialised, regardless of
actual further use. Since the Lzo*Codec classes require the native library to initialise,
the failure is therefore expected.


On Tue, Oct 27, 2015 at 11:42 AM Kiru Pakkirisamy <kirupakkirisamy@yahoo.com> wrote:

I am seeing a weird error after we moved to the new hadoop mapreduce java packages in 2.4

We are not using lzo (as in io.compression.codecs) but we still get this error. Does it mean
we have to have lzo installed even though we are not using ? Thanks.


Regards,

- kiru


2015-10-27 00:18:57,994 ERROR com.hadoop.compression.lzo.GPLNativeCodeLoader | Could not load
native gpl library

java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path

	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886) ~[?:1.7.0_85]

	at java.lang.Runtime.loadLibrary0(Runtime.java:849) ~[?:1.7.0_85]

	at java.lang.System.loadLibrary(System.java:1088) ~[?:1.7.0_85]

	at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:31)
[flow-trunk.242-470787.jar:?]

	at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:60) [flow-trunk.242-470787.jar:?]

	at java.lang.Class.forName0(Native Method) [?:1.7.0_85]

	at java.lang.Class.forName(Class.java:278) [?:1.7.0_85]

	at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1834) [flow-trunk.242-470787.jar:?]

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1799) [flow-trunk.242-470787.jar:?]

	at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
[flow-trunk.242-470787.jar:?]

	at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
[flow-trunk.242-470787.jar:?]

	at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.isSplitable(CombineFileInputFormat.java:159)
[flow-trunk.242-470787.jar:?]

	at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getMoreSplits(CombineFileInputFormat.java:283)
[flow-trunk.242-470787.jar:?]

	at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:243)
[flow-trunk.242-470787.jar:?]

	at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493) [flow-trunk.242-470787.jar:?]


 

Regards,

 - kiru


Mime
View raw message