hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Carey <sc...@richrelevance.com>
Subject Re: [DISCUSS] Remove append?
Date Sat, 24 Mar 2012 02:46:35 GMT


On 3/22/12 5:41 PM, "Eli Collins" <eli@cloudera.com> wrote:

>On Thu, Mar 22, 2012 at 4:27 PM,  <Milind.Bhandarkar@emc.com> wrote:
>> Eli,
>>
>> I think by "current definition of visible length", you mean that once a
>> client opens a file and gets block list, it will always be able to read
>>up
>> to the length at open.
>>
>
>I was thinking of the definition from the design doc. See my last
>comment on HDFS-2288, part of the confusion is that we're using the
>same name for two different things.
>
>> However, correct me if I am wrong, but this definition is already
>> violated, if file is deleted after open.
>
>I think you're right.

Another thing that could be fixed with COW blocks and MVCC principles.  If
a file was opened, then deleted the blocks on the opened file would still
be visible to that client, but no new ones.

>
>> So, truncate does add some complexity, but not a whole lot. If client
>>gets
>> an EOF before length at open, it must retry to see if the new visible
>> length is different (rather than to see if the file does not exist
>> anymore).
>>
>> Right ?
>>
>
>Makes sense. I was thinking you were talking about truncate on open
>files, which be harder.  You can already truncate a file on open, you
>just can't choose the offset you want to truncate at (the NN
>implements this by deleting the file).
>
>Thanks,
>Eli
>
>>
>> ---
>> Milind Bhandarkar
>> Greenplum Labs, EMC
>> (Disclaimer: Opinions expressed in this email are those of the author,
>>and
>> do not necessarily represent the views of any organization, past or
>> present, the author might be affiliated with.)
>>
>>
>>
>> On 3/22/12 4:03 PM, "Eli Collins" <eli@cloudera.com> wrote:
>>
>>>On Thu, Mar 22, 2012 at 3:57 PM, Tsz Wo Sze <szetszwo@yahoo.com> wrote:
>>>>> Do you think having the invariant that blocks are not mutated would
>>>>> significantly simply the design?
>>>>
>>>> No.  As mentioned in my previous email and others, the complexity is
>>>>in
>>>>hflush.  Once we have hflush, append is straightforward.
>>>
>>>I understand that append is a small delta once you have hflush, what
>>>I'm saying is that the overall design of the file system is
>>>significantly simplified if you can assume blocks are not mutated. Eg
>>>see the way truncate is going to interact with the current definition
>>>of visible length (it violates it). Resolving issues like that are
>>>non-trivial.
>>>
>>>Thanks,
>>>Eli
>>>
>>


Mime
View raw message