hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stu24m...@yahoo.com
Subject Re: atomicity of copyFromLocal
Date Tue, 01 Mar 2011 04:51:15 GMT
Pretty sure it's not atomic. I can read files I write via thrift well before they're done.
Rename has always worked for me...

Take care,
-----Original Message-----
From: Ishaaq Chandy <ishaaq@gmail.com>
Date: Tue, 1 Mar 2011 15:22:24 
To: <hdfs-user@hadoop.apache.org>
Reply-To: hdfs-user@hadoop.apache.org
Subject: atomicity of copyFromLocal

Hi all,
How "atomic" is the copyFromLocal call? i.e. on process is in the midst of
uploading a file to HDFS is it possible for another process to start reading
it before the upload is complete?

I am currently safeguarding my code from this possibility by uploading it to
a temporary directory and the renaming it to its final destination (the
assumption being that a rename is "more atomic" than copyFromLocal), but I'd
like to avoid doing this in two steps if possible.


View raw message