hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amareshwari Sriramadasu <amar...@yahoo-inc.com>
Subject Re: How do I include customized InputFormat, InputSplit and RecordReader in a C++ pipes job?
Date Thu, 30 Oct 2008 05:29:02 GMT
Zhengguo 'Mike' SUN wrote:
> Hi, Peeyush,
>
> I guess I didn't make myself clear. I am trying to run a Hadoop pipes job with a combination
of Java classes and C++ classes. So the command I am using is like:
>
> hadoop pipes -conf myconf.xml -inputformat MyInputFormat.class -input in -output out
>
> And it threw ClassNotFoundException for my InputSplit class.
> As I understand "hadoop jar" is used to run a jar file, which is not my case. And there
is a -jar option in "hadoop pipes". But, unfortunately, it is not working for me. So the question
I want to ask is how to include customized Java classes, such as MyInputSplit, in a pipes
job?
>
>   
You are right. -jar option also doesn't add the jar file to classpath on 
the client-side. You can use -libjars option with 0.19. Then, the 
command looks like

hadoop pipes -conf myconf.xml -libjars <jarfile> -inputformat MyInputFormat.class -input
in -output out

I don't see  a way to do this in 0.17.*, one way could be you add it 
explicitly to the classpath on client-side, and add it through the 
option -jar for the job.
Thanks,
Amareshwari
> Thanks,
> Mike
>
>
>
>
> ________________________________
> From: Peeyush Bishnoi <peeyushb@yahoo-inc.com>
> To: core-user@hadoop.apache.org; core-user@hadoop.apache.org
> Sent: Wednesday, October 29, 2008 12:52:18 PM
> Subject: RE: How do I include customized InputFormat, InputSplit and RecordReader in
a C++ pipes job?
>
> Hello Zhengguo ,
>
> Yes , -libjars is the new feature in Hadoop. This feature has been available from Hadoop-0.17.x
, but it is more stable from hadoop 0.18.x 
>
> example to use -libjars...
>
> hadoop jar -libjars <comma seperated list of jars> ...
>
>
> Thanks ,
>
> ---
> Peeyush
>
>
> -----Original Message-----
> From: Zhengguo 'Mike' SUN [mailto:zhengguosun@yahoo.com]
> Sent: Wed 10/29/2008 9:22 AM
> To: core-user@hadoop.apache.org
> Subject: Re: How do I include customized InputFormat, InputSplit and RecordReader in
a C++ pipes job?
>
> Hi, Amareshwari,
>
> Is -libjars a new option in Hadoop 0.19? I am using 0.17.2. The only option I see is
-jar, which didn't work for me. And besides passing them as jar file, is there any other ways
to do that?
>
> Thanks
> Mike
>
>
> ________________________________
> From: Amareshwari Sriramadasu <amarsri@yahoo-inc.com>
> To: core-user@hadoop.apache.org
> Sent: Tuesday, October 28, 2008 11:58:33 PM
> Subject: Re: How do I include customized InputFormat, InputSplit and RecordReader in
a C++ pipes job?
>
> Hi,
>
> How are you passing your classes to the pipes job? If you are passing 
> them as a jar file, you can use -libjars option. From branch 0.19, the 
> libjar files are added to the client classpath also.
>
> Thanks
> Amareshwari
> Zhengguo 'Mike' SUN wrote:
>   
>> Hi,
>>
>> I implemented customized classes for InputFormat, InputSplit and RecordReader in
Java and was trying to use them in a C++ pipes job. The customized InputFormat class could
be included using the -inputformat option, but it threw ClassNotFoundException for my customized
InputSplit class. It seemed the classpath has not been correctly set. Is there any way that
let me include my customized classes in a pipes job?
>>
>>
>>
>>      
>>  
>>     
>
>
>       
>   


Mime
View raw message