hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Hammerton <james.hammer...@mendeley.com>
Subject Re: Unable to read sequence file produced by MR job
Date Wed, 02 Jun 2010 13:08:37 GMT
Hi,

The version is 0.20.1+169.88.

To submit a job I type e.g.:

hadoop jar foo.jar com.mendeley.SomeClass arg1 arg2...

When the classpath is set I've also tried

hadoop com.mendeley.SomeClass arg1 arg2...

I'm using Ubuntu 9.10, and bash is the shell I'm using.

To view a sequence file:

hadoop fs -text foo.seq

When I set the HADOOP_CLASSPATH, I've tried the following variations

export HADOOP_CLASSPATH=foo.jar
export HADOOP_CLASSPATH=foo.jar:$HADOOP_CLASSPATH
export HADOOP_CLASSPATH=foo.jar:/usr/lib/hadoop/lib/
export HADOOP_CLASSPATH=foo.jar:/usr/lib/hadoop-0.20/lib/

Hadoop was installed via the Ubuntu repositories, i.e. using apt-get. It
seems I can either set the HADOOP_CLASSPATH or submit a job but not both.

Regards,

James

On Tue, Jun 1, 2010 at 9:22 PM, Aaron Kimball <aaron@cloudera.com> wrote:

> James,
>
> Which version of Hadoop are you using? HADOOP_CLASSPATH is specifically
> intended to mean additional jars which are added to the list of jars Hadoop
> itself adds to the classpath.
>
> What's the exact command-line you're trying?
>
> - Aaron
>
>
> On Fri, May 28, 2010 at 3:12 PM, James Hammerton <
> james.hammerton@mendeley.com> wrote:
>
>> Thanks.
>>
>> I've found that setting the classpath as you suggested gets the command to
>> work at the expense of the "hadoop jar" command I use to submit jobs no
>> longer finding the hadoop libraries!
>>
>> Even adding the directories onto the classpath along with the .jar file
>> does not fix this. I've worked around it by running the command for viewing
>> sequence files from within a script that sets the classpath. Any ideas how
>> to get both the .jar file and the hadoop libraries into the classpath
>> together so that both job submission and the other commands work?
>>
>> Regards,
>>
>> James
>>
>>
>> On Thu, May 27, 2010 at 7:38 PM, Aaron Kimball <aaron@cloudera.com>wrote:
>>
>>> Put your jar on Hadoop's classpath:
>>>
>>> $ HADOOP_CLASSPATH=path/to/shortdocwritables.jar hadoop fs -text bla
>>> bla....
>>>
>>> - Aaron
>>>
>>>
>>> On Thu, May 27, 2010 at 11:07 AM, James Hammerton <
>>> james.hammerton@mendeley.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I tried using the "hadoop fs -text" command to read a sequence file
>>>> generated by a map reduce job and got the following error:
>>>>
>>>> text: java.io.IOException: WritableName can't load class:
>>>> com.mendeley.clusterer.title.ShortDocWritables
>>>>
>>>> The ShortDocWritables is a Writable I created myself and the
>>>> sequencefile contains these objects. How do I tell this command where to
>>>> find the class?
>>>>
>>>> There was no trouble at all running the map reduce job that produced the
>>>> file.
>>>>
>>>> James
>>>>
>>>> --
>>>> James Hammerton | Senior Data Mining Engineer
>>>> www.mendeley.com/profiles/james-hammerton
>>>>
>>>> Mendeley Limited | London, UK | www.mendeley.com
>>>> Registered in England and Wales | Company Number 6419015
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> James Hammerton | Senior Data Mining Engineer
>> www.mendeley.com/profiles/james-hammerton
>>
>> Mendeley Limited | London, UK | www.mendeley.com
>> Registered in England and Wales | Company Number 6419015
>>
>>
>>
>>
>


-- 
James Hammerton | Senior Data Mining Engineer
www.mendeley.com/profiles/james-hammerton

Mendeley Limited | London, UK | www.mendeley.com
Registered in England and Wales | Company Number 6419015

Mime
View raw message