hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Torsten Curdt <tcu...@apache.org>
Subject Re: rename dir while writing
Date Mon, 05 Nov 2007 18:28:25 GMT
Please note that I am talking about renaming the parent  
directory ...not the file itself!!


On 05.11.2007, at 18:48, Johnson, Jorgen wrote:

> I was under the impression that a file doesn't 'exist' in hdfs  
> global namespace until it has been closed, and that appends are not  
> supported (yet?).  So this situation, at current time, should not  
> be possible, correct?
> I remember seeing a jira issue where someone wanted to change it so  
> files show up before they are closed, but I didn't catch what the  
> resolution was for that issue.  Am I missing something here?
> -jorgenj
> -----Original Message-----
> From: Owen O'Malley [mailto:oom@yahoo-inc.com]
> Sent: Monday, November 05, 2007 9:24 AM
> To: hadoop-user@lucene.apache.org
> Subject: Re: rename dir while writing
> On Nov 5, 2007, at 2:05 AM, Torsten Curdt wrote:
>> Is there anywhere documented the expected behavior of concurrent
>> changes in the filesystem? As an example:
>> Hdfs client C1 is slowly writing to "/path/a/file". Now another
>> hdfs client C2 renames "/path/a" to /path/b".
>> What happens? Will C1 continue to write but the file will be in "/
>> path/b" when it closes the file? Will C2 get an exception because
>> C1 is writing to "/path/a" so it cannot be renamed? And so on...
>> I will probably run off an do some test myself ...but is this
>> documented somewhere? I couldn't find it.
> The short answer is that currently bad things happen. All of the hdfs
> protocols call files by name and so changing the name of the file
> while someone is writing to it will break the writer. In my opinion,
> it would be better if dfs.ClientProtocol.create returned a lease
> handle, which was used for the follow up addBlock, complete, and
> renewLease.
> -- Owen

View raw message