Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <bec373e1002020909p970eb0bv7aed70949df81cb2@mail.gmail.com>
References: <bec373e1002020909p970eb0bv7aed70949df81cb2@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 2 Feb 2010 11:41:19 -0800
Message-ID: <45f85f71002021141o1dc81dd8p1cefefdf9d853d81@mail.gmail.com>
Subject: Re: ClassCastException in lzo indexer
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502af825af687047ea34a3e

--00504502af825af687047ea34a3e
Content-Type: text/plain; charset=ISO-8859-1

Hi Vasilis,

Did you make sure to "ant clean" before rebuilding hadoop-lzo if you updated
the code? Also, can you paste your configuration for io.compression.codecs ?

Thanks
-Todd

On Tue, Feb 2, 2010 at 9:09 AM, Vasilis Liaskovitis <vliaskov@gmail.com>wrote:

> Hi,
>
> I am trying to use hadoop-0.20.1 and hadoop-lzo
> (http://github.com/kevinweil/hadoop-lzo) to index an lzo file. I 've
> followed the instructions and copied both jar and native libs in my
> classpaths. I am getting this error in both local and distributed
> indexer mode
>
> bin/hadoop jar lib/hadoop-lzo-0.3.0.jar
> com.hadoop.compression.lzo.LzoIndexer /data/userVisits.lzo
>
> 10/02/02 17:30:38 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
> 10/02/02 17:30:38 INFO lzo.LzoCodec: Successfully loaded & initialized
> native-lzo library
> 10/02/02 17:30:38 INFO lzo.DistributedLzoIndexer: Adding LZO file
> /data/UserVisits.lzo to indexing list (no index currently exists)
> 10/02/02 17:30:38 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the
> same.
> 10/02/02 17:30:39 INFO input.FileInputFormat: Total input paths to process
> : 1
> 10/02/02 17:30:39 INFO mapred.JobClient: Running job: job_201002020748_0409
> 10/02/02 17:30:40 INFO mapred.JobClient:  map 0% reduce 0%
> 10/02/02 17:31:02 INFO mapred.JobClient: Task Id :
> attempt_201002020748_0409_m_000000_0, Status : FAILED
> java.lang.ClassCastException:
> com.hadoop.compression.lzo.LzopCodec$LzopDecompressor cannot be cast
> to com.hadoop.compression.lzo.LzopDecompressor
>
> same error for local indexer:
>
> user2@amdqc08:~/hadoop-0.20.1-prof> bin/hadoop jar
> lib/hadoop-lzo-0.3.0.jar com.hadoop.compression.lzo.LzoIndexer
> /data/UserVisits.lzo
> 10/02/02 17:38:47 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
> 10/02/02 17:38:47 INFO lzo.LzoCodec: Successfully loaded & initialized
> native-lzo library
> 10/02/02 17:38:47 INFO lzo.LzoIndexer: [INDEX] LZO Indexing file
> /data/UserVisits.lzo, size 9.94 GB...
> Exception in thread "main" java.lang.ClassCastException:
> com.hadoop.compression.lzo.LzopCodec$LzopDecompressor cannot be cast
> to com.hadoop.compression.lzo.LzopDecompressor
>
> is hadoop-0.20.1 compatible with the git/master of hadoop-lzo? Or do I
> need to use some older version of hadoop-lzo to be compatible with
> hadoop-0.20.1?
>
> - a different, but relevant question: In order to compress
> intermediate map outputs with lzo and process them in an efficient
> way, does the map/reduce job need to explicitly create index files for
> the compressed intermediate files? I think that at this shuffle stage,
> input files have already been split and we are not relying on indexing
> the intermediate lzo files for parallel shuffling. Is that correct? Or
> would a job need to index the intermediate files? If yes, can this be
> handled in an automatic fashion by hadoop-lzo?
>
> any suggestions are welcome.
> thanks,
>
> - Vasilis
>

--00504502af825af687047ea34a3e--