Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of mayingnanapple@gmail.com
 designates 209.85.210.48 as permitted sender)
MIME-Version: 1.0
Date: Wed, 23 May 2012 18:29:57 +0800
Message-ID: 
 <CA+3Vhvj5w4mitPd_XVmDyUufGhhNzSZZbjp3Xxkaj4o8O66wAg@mail.gmail.com>
Subject: Hadoop LZO compression
From: Yingnan Ma <mayingnanapple@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=e89a8ff24e11dcda7304c0b19dee

--e89a8ff24e11dcda7304c0b19dee
Content-Type: text/plain; charset=ISO-8859-1

 Hi,

I encounter a problem about when I install the LZO,
after i install it, I found that it can run on Pig scripts
and streaming scripts and when I check these jobs though
 jobtracker , it shows that
*mapred.compress.map.output *true,
*io.compression.codecs *
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec

*io.compression.codec.lzo.class*com.hadoop.compression.lzo.LzoCodec

and the pig && streaming can run also.

However, I found that it would gave me some error notices

such as:


2012-05-23 17:32:57,052 [Thread-6] ERROR
com.hadoop.compression.lzo.GPLNativeCodeLoader - Could not load native
gpl library
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1738)
        at java.lang.Runtime.loadLibrary0(Runtime.java:823)
        at java.lang.System.loadLibrary(System.java:1028)
        at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:32)
        at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:71)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943)
        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:89)
        at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
        at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:46)
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:254)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
        at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944)
        at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961)
        at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
        at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
        at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
        at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
        at java.lang.Thread.run(Thread.java:662)
2012-05-23 17:32:57,053 [Thread-6] ERROR
com.hadoop.compression.lzo.LzoCodec - Cannot load native-lzo without
native-hadoop
2012-05-23 17:32:57,070 [Thread-6] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths (combined) to process : 1
2012-05-23 17:32:58,419 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201204051731_1249
2012-05-23 17:32:58,419 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://hdjt:50030/jobdetails.jsp?jobid=job_201204051731_1249
2012-05-23 17:33:14,526 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 50% complete
2012-05-23 17:33:18,134 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2012-05-23 17:33:18,136 [main] INFO
org.apache.pig.tools.pigstats.PigStats - Script Statistics:

HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt      Features
0.20.2-cdh3u0   0.8.0-cdh3u0    root    2012-05-23 17:32:54
2012-05-23 17:33:18    FILTER
 Success!
 Job Stats (time in seconds):
JobId   Maps    Reduces MaxMapTime      MinMapTIme      AvgMapTime
 MaxReduceTime  MinReduceTime   AvgReduceTime   Alias   Feature
Outputs
job_201204051731_1249   1       0       10      10      10      0
 0     A,B      MAP_ONLY
hdfs://hdmaster:54310/tmp/temp-1842686846/tmp-2027515206,

It make me confuse because if it has some issues, it would not work,
however it may work. So I need some help, thank you for your help!

Best Regards

Malone


2012-05-23

--e89a8ff24e11dcda7304c0b19dee--