hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: MultipleOutputs support
Date Tue, 02 Aug 2011 05:30:31 GMT
Hello Vanja,

The CDH report is best submitted to cdh-user@cloudera.org where an
action could then be taken. Would help if you can describe your new
API MO issue as well!

Regarding your general multiple outputs in output directory issue,
check this FAQ to get a full understanding of how task committing can
help: http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2BAC8-write-to_hdfs_files_directly_from_map.2BAC8-reduce_tasks.3F

Basically, you have to write to the task attempts' working directory.
This way, even speculatives are OK to have, and only successfully
completed (and 'committed', like in a DB) tasks can get to push the
final output after the job completes, and the rest are cleaned away
from the temporary working directory.

Hope this helps! Although, I consider MO to be painless and the true
way to go to do multiple outputs in MR right now. So would like to
help to see if we can fix those issues up as well.

On Tue, Aug 2, 2011 at 3:51 AM, Vanja Komadinovic <vanjakom@gmail.com> wrote:
> Hi all,
> I'm trying to create M/R tasks that will output more than one "type" of data. Ideal thing
would be MultipleOutputs feature of Map Reduce, but in our current production version, CDH3
( 0.20.2 ), this support is broken.
> So, I tried to simulate MultipleOutputs. In Reducer setup I'm opening hdfs output stream,
during reduce calls writing to stream and in close call closing stream. Output streams are
named with attempt id inside. This is working great. Speculative execution is disabled, but
sometimes one of reduce task fail, and I' getting two files for reducer on same data. Is there
any way to find out which task attempts where successful, so I can delete unneeded data after
successful job? I'm using new MapReduce API. Or some other better idea to achieve this?
> Best,
> Vanja
> Komadinovic Vanja
> +381 (64) 296 03 43
> vanjakom@gmail.com

Harsh J

View raw message