hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Radim Kolar <...@filez.com>
Subject Re: When speculative execution is true, there is a data loss issue with multpleoutputs
Date Wed, 21 Nov 2012 15:44:24 GMT
this is another problem with fileoutputformat committer, its related to 


it works like this: if multipleoutput is relative to job output, then 
there is a workaround to make it work with commiter and outputs from 
multiple tasks do not clash with each other, problem mentioned in ticket 
cheats that relative vs absolute output path detection and all output is 
lost on task commit.

But if output is absolute path, then its written directly to output file 
which fails because writers from multiple attempts crash together.

View raw message