Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <4f10e2890902141446n7b6ab23evee18199f338e505@mail.gmail.com>
References: <4f10e2890902141446n7b6ab23evee18199f338e505@mail.gmail.com>
Date: Sat, 14 Feb 2009 15:45:31 -0800
Message-ID: <e4eb02ce0902141545j233bd847p9f08602c593846ee@mail.gmail.com>
Subject: Re: Race Condition?
From: Matei Zaharia <matei@cloudera.com>
To: core-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163646c7768b3d440462e98cda

--00163646c7768b3d440462e98cda
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Have you logged the output of the dfs command to see whether it's always
succeeded the copy?

On Sat, Feb 14, 2009 at 2:46 PM, S D <sd.codewarrior@gmail.com> wrote:

> In my Hadoop 0.19.0 program each map function is assigned a directory
> (representing a data location in my S3 datastore). The first thing each map
> function does is copy the particular S3 data to the local machine that the
> map task is running on and then being processing the data; e.g.,
>
> command = "hadoop dfs -copyToLocal #{s3dir} #{localdir}"
> system "#{command}"
>
> In the above, "s3dir" is a directory that creates "localdir" - my
> expectation is that "localdir" is created in the work directory for the
> particular task attempt. Following this copy command I then run a function
> that processes the data; e.g.,
>
> processData(localdir)
>
> In some instances my map/reduce program crashes and when I examine the logs
> I get a message saying that "localdir" can not be found. This confuses me
> since the hadoop shell command above is blocking so that localdir should
> exist by the time processData() is called. I've found that if I add in some
> diagnostic lines prior to processData() such as puts statements to print
> out
> variables, I never run into the problem of the localdir not being found. It
> is almost as if localdir needs time to be created before the call to
> processData().
>
> Has anyone encountered anything like this? Any suggestions on what could be
> wrong are appreciated.
>
> Thanks,
> John
>

--00163646c7768b3d440462e98cda--