hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Hammerton <james.hammer...@mendeley.com>
Subject Re: Unable to read sequence file produced by MR job
Date Fri, 28 May 2010 14:12:58 GMT
Thanks.

I've found that setting the classpath as you suggested gets the command to
work at the expense of the "hadoop jar" command I use to submit jobs no
longer finding the hadoop libraries!

Even adding the directories onto the classpath along with the .jar file does
not fix this. I've worked around it by running the command for viewing
sequence files from within a script that sets the classpath. Any ideas how
to get both the .jar file and the hadoop libraries into the classpath
together so that both job submission and the other commands work?

Regards,

James

On Thu, May 27, 2010 at 7:38 PM, Aaron Kimball <aaron@cloudera.com> wrote:

> Put your jar on Hadoop's classpath:
>
> $ HADOOP_CLASSPATH=path/to/shortdocwritables.jar hadoop fs -text bla
> bla....
>
> - Aaron
>
>
> On Thu, May 27, 2010 at 11:07 AM, James Hammerton <
> james.hammerton@mendeley.com> wrote:
>
>> Hi,
>>
>> I tried using the "hadoop fs -text" command to read a sequence file
>> generated by a map reduce job and got the following error:
>>
>> text: java.io.IOException: WritableName can't load class:
>> com.mendeley.clusterer.title.ShortDocWritables
>>
>> The ShortDocWritables is a Writable I created myself and the sequencefile
>> contains these objects. How do I tell this command where to find the class?
>>
>> There was no trouble at all running the map reduce job that produced the
>> file.
>>
>> James
>>
>> --
>> James Hammerton | Senior Data Mining Engineer
>> www.mendeley.com/profiles/james-hammerton
>>
>> Mendeley Limited | London, UK | www.mendeley.com
>> Registered in England and Wales | Company Number 6419015
>>
>>
>>
>>
>


-- 
James Hammerton | Senior Data Mining Engineer
www.mendeley.com/profiles/james-hammerton

Mendeley Limited | London, UK | www.mendeley.com
Registered in England and Wales | Company Number 6419015

Mime
View raw message