hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Heidemann <jo...@ISI.EDU>
Subject Re: Using hadoop for distributed rendering
Date Wed, 17 Oct 2007 21:43:19 GMT
On Tue, 16 Oct 2007 16:37:41 BST, Robert Jessop wrote: 
>Hi there. I did a search of the mailing list archives looking for
>something similar to this, but I didn't find anything so apologies if
>this has been discussed before.
>I'm investigating using Hadoop for distributed rendering. The Mapper
>would define the tiles to be rendered and the nodes would render them
>using the scene data (which is for the sake of argument, all wrapped
>up in one big binary file on the HDFS). The reducer would take the
>output tiles and stitch them together to form the final image. I have
>a system that does this already but it doesn't have any of the
>advantages of a distributed file system, there are lots of IO
>bottlenecks and communication overheads. All the code is currently in
>Does this sound like a good use for Hadoop?

We did almost exactly this with our whole Internet map

By "almost": we rendered the tiles in the reduce phase, but wrote them
out-of-band (direct to the file system), rather than as direct reduce
output.  Since we were writing pngs, I didn't want to have to
decapsulate them from the reducer output stream.
We were using hadoop streaming; perhaps this goes away if we had
directly written a Java reducer with custom output, but that seems

We then did the stitching together on a single workstation after the fact.

About 2.8 billion records in through a custom map IO function, with ~43k
tiles output over ~19 hours on 52 cores.  Then maybe 4 hours stiching
things together on a single box, and ~36 hours printing on a single

(By the way, if anyone knows how to stream huge images at an HP
Designjet 800PS printer without blowing up memory, please let me know.
The postscript redbook commands to control paper advance and non-cutting
seem to be ignored.)

You don't say what kind of rendering you're doing.  If it's movie-style
3-D rendering, I expect you'd need some work to get benefit from HDFS or
other file systems---the input needed for rendering (compositing,
textures, models, etc.) is not an obvious fit for map/reduce, at least
to me.

   -John Heidemann

View raw message