hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Kimball <aa...@cloudera.com>
Subject Re: Unable to read sequence file produced by MR job
Date Tue, 01 Jun 2010 20:22:45 GMT
James,

Which version of Hadoop are you using? HADOOP_CLASSPATH is specifically
intended to mean additional jars which are added to the list of jars Hadoop
itself adds to the classpath.

What's the exact command-line you're trying?

- Aaron

On Fri, May 28, 2010 at 3:12 PM, James Hammerton <
james.hammerton@mendeley.com> wrote:

> Thanks.
>
> I've found that setting the classpath as you suggested gets the command to
> work at the expense of the "hadoop jar" command I use to submit jobs no
> longer finding the hadoop libraries!
>
> Even adding the directories onto the classpath along with the .jar file
> does not fix this. I've worked around it by running the command for viewing
> sequence files from within a script that sets the classpath. Any ideas how
> to get both the .jar file and the hadoop libraries into the classpath
> together so that both job submission and the other commands work?
>
> Regards,
>
> James
>
>
> On Thu, May 27, 2010 at 7:38 PM, Aaron Kimball <aaron@cloudera.com> wrote:
>
>> Put your jar on Hadoop's classpath:
>>
>> $ HADOOP_CLASSPATH=path/to/shortdocwritables.jar hadoop fs -text bla
>> bla....
>>
>> - Aaron
>>
>>
>> On Thu, May 27, 2010 at 11:07 AM, James Hammerton <
>> james.hammerton@mendeley.com> wrote:
>>
>>> Hi,
>>>
>>> I tried using the "hadoop fs -text" command to read a sequence file
>>> generated by a map reduce job and got the following error:
>>>
>>> text: java.io.IOException: WritableName can't load class:
>>> com.mendeley.clusterer.title.ShortDocWritables
>>>
>>> The ShortDocWritables is a Writable I created myself and the sequencefile
>>> contains these objects. How do I tell this command where to find the class?
>>>
>>> There was no trouble at all running the map reduce job that produced the
>>> file.
>>>
>>> James
>>>
>>> --
>>> James Hammerton | Senior Data Mining Engineer
>>> www.mendeley.com/profiles/james-hammerton
>>>
>>> Mendeley Limited | London, UK | www.mendeley.com
>>> Registered in England and Wales | Company Number 6419015
>>>
>>>
>>>
>>>
>>
>
>
> --
> James Hammerton | Senior Data Mining Engineer
> www.mendeley.com/profiles/james-hammerton
>
> Mendeley Limited | London, UK | www.mendeley.com
> Registered in England and Wales | Company Number 6419015
>
>
>
>

Mime
View raw message