hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Luya <alexander.l...@gmail.com>
Subject Re: How to get lzo compression library loaded?
Date Sun, 01 Aug 2010 06:15:39 GMT
Hi,
    what I am trying to run is:
---------------------------------------------------------------------------------------------------------------------------
hadoop jar /usr/local/hadoop/hadoop-0.20.2/lib/hadoop-lzo-0.4.4.jar 
com.hadoop.compression.lzo.DistributedLzoIndexer target.lzo
---------------------------------------------------------------------------------------------------------------------------

env | grep -i hadoop
---------------------------------------------------------------------------------------------------------------------------
NUTCH_HOME=/usr/local/hadoop/nutch-1.1
HADOOP_HOME=/usr/local/hadoop/hadoop-0.20.2
HBASE_HOME=/usr/local/hadoop/hbase-0.20.4
PATH=/media/Work/workspace/HDScript:/usr/local/hadoop/nutch-1.1/bin:/media/Backup/Hive/hive-0.5.0-
dev/bin:/usr/local/hadoop/zookeeper-3.3.1/bin:/usr/local/hadoop/hbase-0.20.4/bin:/usr/local/hadoop/nutch-1.1/bin:/opt/hypertable/0.9.3.4/bin:/usr/local/hadoop/hadoop-0.20.2/bin:/home/alex/jetty-
hightide-7.1.5.v20100705/bin:/usr/local/hadoop/jdk1.6.0_20/bin:/usr/local/hadoop/jdk1.6.0_20/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
PWD=/usr/local/hadoop/hadoop-0.20.2/conf
JAVA_HOME=/usr/local/hadoop/jdk1.6.0_20
ZOOKEEPER_HOME=/usr/local/hadoop/zookeeper-3.3.1
OLDPWD=/usr/local/hadoop/hadoop-0.20.2

---------------------------------------------------------------------------------------------------------------------------


env | grep -i classpath
---------------------------------------------------------------------------------------------------------------------------
no output
---------------------------------------------------------------------------------------------------------------------------


All my operations are:
1, got source code from http://github.com/kevinweil/hadoop-lzo,compiled 
them successfully,and then
2,,copy hadoop-lzo-0.4.4.jar to directory:$HADOOP_HOME/lib of each master and 
slave
3,Copy all files under directory:../Linux-amd64-64/lib to directory:
$HADDOOP_HOME/lib/native/Linux-amd64-64 of each master and slave
4,and upload a file:test.lzo to HDFS
5,then run:hadoop jar $HADOOP_HOME/lib/hadoop-lzo-0.4.4.jar 
com.hadoop.compression.lzo.DistributedLzoIndexer test.lzo to test

is any other configuration needed?

On Sunday, August 01, 2010 11:37:19 am Alex Kozlov wrote:
> Hi Alex,
> 
> org.apache.hadoop.io.compress.GzipCodec is in the hadoop-core-*.jar.  You
> don't need to add /usr/local/hadoop/hadoop-0.20.2/lib/ to the classpath
> since hadoop shell script does it for you.  What is exactly the command you
> are trying to run?  Can you also give the output of 'env | grep -i hadoop'
> and 'env | grep -i classpath'.
> 
> Alex K
> 
> On Sat, Jul 31, 2010 at 1:38 AM, Alex Luya <alexander.luya@gmail.com> wrote:
> > Hello:
> >    I have followed this link:http://code.google.com/p/hadoop-gpl-
> > 
> > compression/wiki/FAQ<http://code.google.com/p/hadoop-gpl-%0Acompression/w
> > iki/FAQ>to install lzo compression library,and copy hadoop-lzo-0.4.4.jar
> > to $HADOOP_HOME/lib,and all files under
> > ..lib/native/Linux-amd64-64 to $HADOOP_HOME/lib/native/Linux-amd64-64,
> > and run example,but got this errors:
> > 
> > -------------------------------------------------------------------------
> > --------------------------------------- Exception in thread "main"
> > java.lang.IllegalArgumentException: Compression codec
> > 
> >                org.apache.hadoop.io.compress.GzipCodec not found.
> >        
> >        at
> > 
> > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(Com
> > pressionCodecFactory.java:96)
> > 
> >        at
> > 
> > org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionC
> > odecFactory.java:134)
> > 
> >        at
> > 
> > com.hadoop.compression.lzo.LzoIndex.createIndex(LzoIndex.java:202)
> > 
> >        at
> > 
> > com.hadoop.compression.lzo.LzoIndexer.indexSingleFile(LzoIndexer.java:117
> > )
> > 
> >        at
> > 
> > com.hadoop.compression.lzo.LzoIndexer.indexInternal(LzoIndexer.java:98)
> > 
> >        at com.hadoop.compression.lzo.LzoIndexer.index(LzoIndexer.java:52)
> >        at com.hadoop.compression.lzo.LzoIndexer.main(LzoIndexer.java:137)
> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >        at
> > 
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
> > :39)
> > 
> >        at
> > 
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
> > mpl.java:25)
> > 
> >        at java.lang.reflect.Method.invoke(Method.java:597)
> >        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > 
> > Caused by: java.lang.ClassNotFoundException:
> >                org.apache.hadoop.io.compress.GzipCodec
> >        
> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >        at java.lang.Class.forName0(Native Method)
> >        at java.lang.Class.forName(Class.java:247)
> >        at
> > 
> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:76
> > 2)
> > 
> >        at
> > 
> > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(Com
> > pressionCodecFactory.java:89)
> > 
> >        ... 11 more
> > 
> > -------------------------------------------------------------------------
> > ---------------------------------------
> > 
> > and then I try to add this to hadoop-env.sh:
> > 
> > export
> > HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/local/hadoop/hadoop-0.20.2/lib/
> > 
> > Same problem as before,this problem is killing me,because it has
> > surrounded me
> > for one month.

Mime
View raw message