hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Groschupf ...@media-style.com>
Subject Re: IdentityMapper
Date Thu, 20 Apr 2006 13:52:37 GMT
Hi Doug,

> I don't understand the problem here.

There is no really problem, just a question to better understand hadoop.
My real problem is that the map  and reduce task have to have the  
same key and value class.
Since changing this is a little bit more work as far I can say that,  
I was thinking having one job that do my map task with these key -  
value classes and having another job doing my reduce job with a  
different key - value class would be a good workaround.

> Some map function is required for any data to make it to reduce.   
> IdentityMapper simply copies all map input without altering it.
But required to have the same key - value classes.
If you take a look to my Black White List patch for nutch, you will  
find a FormatConverter that is only required since the problem with  
key value classes.
In case no Mapping at all would be required and the input is just  
copied without sending it to any writer again.
The reducing can be done on the input of a job.

Sorry I hope this makes more sense and  I could clarify my question.


View raw message