hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Isaacson <...@cloudera.com>
Subject Re: Detect when file is not being written by another process
Date Tue, 25 Sep 2012 20:15:26 GMT
On Tue, Sep 25, 2012 at 9:28 AM, Peter Sheridan
<psheridan@millennialmedia.com> wrote:
> We're using Hadoop 1.0.3.  We need to pick up a set of large (4+GB) files
> when they've finished being written to HDFS by a different process.

The common way to solve this problem is to modify the writing
application to write to a temporary filename and then rename the
temporary to the target filename when the write is complete.

That way, if the file exists without the temporary tag, the reader can
be confident the file is complete.


View raw message