hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Torsten Curdt <tcu...@apache.org>
Subject Re: rename dir while writing
Date Mon, 05 Nov 2007 18:28:25 GMT
Please note that I am talking about renaming the parent  
directory ...not the file itself!!

cheers
--
Torsten

On 05.11.2007, at 18:48, Johnson, Jorgen wrote:

> I was under the impression that a file doesn't 'exist' in hdfs  
> global namespace until it has been closed, and that appends are not  
> supported (yet?).  So this situation, at current time, should not  
> be possible, correct?
>
> I remember seeing a jira issue where someone wanted to change it so  
> files show up before they are closed, but I didn't catch what the  
> resolution was for that issue.  Am I missing something here?
>
> -jorgenj
>
> -----Original Message-----
> From: Owen O'Malley [mailto:oom@yahoo-inc.com]
> Sent: Monday, November 05, 2007 9:24 AM
> To: hadoop-user@lucene.apache.org
> Subject: Re: rename dir while writing
>
>
> On Nov 5, 2007, at 2:05 AM, Torsten Curdt wrote:
>
>> Is there anywhere documented the expected behavior of concurrent
>> changes in the filesystem? As an example:
>>
>> Hdfs client C1 is slowly writing to "/path/a/file". Now another
>> hdfs client C2 renames "/path/a" to /path/b".
>>
>> What happens? Will C1 continue to write but the file will be in "/
>> path/b" when it closes the file? Will C2 get an exception because
>> C1 is writing to "/path/a" so it cannot be renamed? And so on...
>>
>> I will probably run off an do some test myself ...but is this
>> documented somewhere? I couldn't find it.
>
> The short answer is that currently bad things happen. All of the hdfs
> protocols call files by name and so changing the name of the file
> while someone is writing to it will break the writer. In my opinion,
> it would be better if dfs.ClientProtocol.create returned a lease
> handle, which was used for the follow up addBlock, complete, and
> renewLease.
>
> -- Owen


Mime
View raw message