hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erez Katz <erez_k...@yahoo.com>
Subject Re: custom partitioner with streaming
Date Tue, 16 Mar 2010 01:05:44 GMT
Ok, Looking/poking at the classes inside streaming.jar I noticed that the partitioner class
is loaded from the same class loader that loaded streaming.jar.

A quick and dirty experiment where I put my custom partitioner class in streaming.jar worked.

Thing is, I'd hate to rebuild standard jars that come with hadoop.
The only way to start a mapreduce job that depends on multiple jar files is using the -libjars
parameter, but it is not supported with streaming....

So I ask again - any way to submit a custom partitioner class to a streaming applications?



--- On Mon, 3/8/10, Erez Katz <erez_katz@yahoo.com> wrote:

From: Erez Katz <erez_katz@yahoo.com>
Subject: Re: custom partitioner with streaming
To: mapreduce-dev@hadoop.apache.org
Date: Monday, March 8, 2010, 4:50 PM

Anyone?

--- On Fri, 3/5/10, Erez Katz <erez_katz@yahoo.com> wrote:

> From: Erez Katz <erez_katz@yahoo.com>
> Subject: custom partitioner with streaming
> To: mapreduce-dev@hadoop.apache.org
> Date: Friday, March 5, 2010, 10:15 AM
> Hi,
> 
> I have a python based map reduce application.
> 
> I would like to define my own paritioner, (just like I
> would have done with pipes/java).
> 
> How do I specify the jar file that contains my custom
> partitioner, assuming it is written in java... can a
> paritioner be written in python/ruby?
> 
> Thanks,
> 
>   Erez Katz
> 
> p.s.
> I am using hadoop 0.19.1
> 
> 
>       
> 






      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message