Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of adi@cloudera.com designates
 209.85.160.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <4A3B3466BCAEF24E80F8EB422B1EE0010F011593@MBX021-E3-NJ-6.exch021.domain.local>
References: 
 <4A3B3466BCAEF24E80F8EB422B1EE0010F011593@MBX021-E3-NJ-6.exch021.domain.local>
Date: Tue, 25 Sep 2012 13:15:26 -0700
Message-ID: 
 <CAE_FxDgAo3xw9rE21p=XVQY1uETprn8Jn5+iV0zze_Z7OwHwfQ@mail.gmail.com>
Subject: Re: Detect when file is not being written by another process
From: Andy Isaacson <adi@cloudera.com>
To: user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Sep 25, 2012 at 9:28 AM, Peter Sheridan
<psheridan@millennialmedia.com> wrote:
> We're using Hadoop 1.0.3.  We need to pick up a set of large (4+GB) files
> when they've finished being written to HDFS by a different process.

The common way to solve this problem is to modify the writing
application to write to a temporary filename and then rename the
temporary to the target filename when the write is complete.

That way, if the file exists without the temporary tag, the reader can
be confident the file is complete.

-andy