crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Narlin M <>
Subject Crunch DoFn vs Mapper/reducer
Date Wed, 14 Aug 2013 19:59:27 GMT
I have just recently started using Crunch, having been recommended to use
it instead of writing plain map reduce jobs. As I was going through the
crunch documentation, some questions came to my mind. Am I correct in
saying that the DoFn family of functions will internally spawn map-reduce
jobs, so there is no need to write separate mapper or reducer classes? If
so, I agree that this will abstract some of the lower level details from
the programmer, but at the same time, does it not lower the programmer's
control over the processing logic?

Also, will there be situations when separate mapper / reducer classes will
be required in addition to the DoFn functions?


View raw message