hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Anderson <...@monkey.org>
Subject Re: streaming question
Date Thu, 18 Sep 2008 21:12:33 GMT

On 16-Sep-08, at 1:25 AM, Christian Ulrik S√łttrup wrote:

> Ok i've tried what you suggested and all sorts of combinations with  
> no luck.
> Then I went through the source of the Streaming lib. It looks like  
> it checks for the existence
> of the combiner while it is building the jobconf i.e. before the job  
> is sent to the nodes.
> It calls class.forName() on the combiner in goodClassOrNull() from  
> StreamUtil.java
> called from setJobconf() in StreamJob.java.
> Anybody have an idea how i can use a custom combiner? would I have  
> to package it into the streaming jar?

That's what the streaming docs say you have to do - make your own  
streaming jar with them included.  I tried the cache and jar arguments  
myself once, and Hadoop wasn't able to find them to use for the  
framework hooks, even when my streaming executables themselves were  
able to find them.

Karl Anderson

View raw message