hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "W.P. McNeill" <bill...@gmail.com>
Subject Where should combiner processes write temporary sequence files?
Date Mon, 09 May 2011 21:19:05 GMT
I have a reducer that creates a temporary sequence file.  I am generating
this file by calling SequenceFileOutputFormat.getPathForWorkFile() and
passing the result into SequenceFile.createWriter().  Now I also want to use
the same reducer code as a combiner.
SequenceFileOutputFormat.getPathForWorkFile()
won't work because a combiner does not have access to the job output
directory.

What is the right way to automatically generate a non-colliding temporary
sequence file that can be used by both a combiner and a reducer?

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message