Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates
 209.85.214.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <505111F9.6090608@filez.com>
References: <505111F9.6090608@filez.com>
From: Harsh J <harsh@cloudera.com>
Date: Thu, 13 Sep 2012 09:00:22 +0530
Message-ID: 
 <CAOcnVr3yOiFt2sx5T8Sdyp9+_6weaGXmd4_yGadQJqcCbymGfA@mail.gmail.com>
Subject: Re: multipleoutputs does not like speculative execution in map-only
 job
To: user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hey Radim,

Does your job use the FileOutputCommitter?

On Thu, Sep 13, 2012 at 4:21 AM, Radim Kolar <hsn@filez.com> wrote:
> with speculative execution enabled Hadoop can run task attempt on more then
> 1 node. If mapper is using multipleoutputs then second attempt (or sometimes
> even all) fails to create output file because it is being created by another
> attempt:
>
> attempt_1347286420691_0011_m_000000_0
> attempt_1347286420691_0011_m_000000_1
> ..
> fails with
> Error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed
> to create file /cznewgen/segments/20120907190053/parse_db/-m-00000
>
> in my code i am using mos.write with 4 arguments. this problem is discussed
> in javadoc for FileOutputFormat function getWorkOutputPath, its possible to
> change MultipleOutputs to take advantage of this function?
>
> or its better to change FileOoutputFormat.getUniqueFile() to append last
> digit in attempt id to filename to create unique names such as
> /cznewgen/segments/20120907190053/parse_db/-m-00000_0 ?


-- 
Harsh J