hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: Poly-reduce?
Date Fri, 24 Aug 2007 16:53:49 GMT
Ted Dunning wrote:
> It isn't hard to implement these programs as multiple fully fledged
> map-reduces, but it appears to me that many of them would be better
> expressed as something more like a map-reduce-reduce program.
> 
> [ ... ]
> 
> Expressed conventionally, this would have write all of the user sessions to
> HDFS and a second map phase would generate the pairs for counting.  The
> opportunity for efficiency would come from the ability to avoid writing
> intermediate results to the distributed data store.
>     
> Has anybody looked at whether this would help and whether it would be hard
> to do?

It would job tracker more complicated, and might not help job execution 
time that much.

Consider implementing this as multiple map reduce steps, but using a 
replication level of one for intermediate data.  That would mostly have 
the performance characteristics you want.  But if a node died, things 
could not intelligently automatically re-create just the missing data. 
Instead the application would have to re-run the entire job, or subsets 
of it, in order to re-create the un-replicated data.

Under poly-reduce, if a node failed, all tasks that were incomplete on 
that node would need to be restarted.  But first, their input data would 
need to be located.  If you saved all intermediate data in the course of 
a job (which would be expensive) then the inputs that need re-creation 
would mostly just be those that were created on the failed node.  But 
this failure would generally cascade all the way back to the initial map 
stage.  So a single machine failure in the last phase could double the 
run time of the job, with most of the cluster idle.

If, instead, you used normal mapreduce, with intermediate data 
replicated in the filesystem, a single machine failure in the last phase 
would only require re-running tasks from the last job.

Perhaps, when chaining mapreduces, one should use a lower replication 
level for intermediate data, like two.  Additionally, one might wish to 
relax the one-replica-off-rack criterion for such files, so that 
replication is faster, and since whole-rack failures are rare.  This 
might give good chained performance, but keep machine failures from 
knocking tasks back to the start of the chain.  Currently its not 
possible to disable the one-replica-off-rack preference, but that might 
be a reasonable feature request.

Doug


Mime
View raw message