hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vidur Goyal" <vi...@students.iiit.ac.in>
Subject Re: Overwriting the same block instead of creating a new one
Date Tue, 22 Jun 2010 05:42:28 GMT
Like in any other filesystem as ext4 , in case of overwrite why don't we
update the existing physical memory. Why is there a need to allocate
memory every time when an overwrite takes place. Isn't this a overhead.


> I know about the current behaviour of HDFS. I am proposing this new
> behaviour which i mentioned in my first mail.
>
> In Hadoop-0.20.2 , a new block is allocated and stored at datanodes and a
> new INode is created in namespace. Why is an overwrite considered as a
> file creation operation.
>
> -vidur
>> Hi Vidur,
>>
>> I'm not following. The "overwrite" flag causes the file to be
>> overwritten
>> starting at offset 0 - it doesn't allow you to retain any bit of the
>> preexisting file. It's equivalent to a remove followed by a create.
>> Think
>> of
>> it like O_TRUNC.
>>
>> -Todd
>>
>> On Mon, Jun 21, 2010 at 10:03 PM, Vidur Goyal
>> <vidur@students.iiit.ac.in>wrote:
>>
>>> Dear Todd,
>>>
>>> By truncating i meant removing unused *blocks* from the namespace and
>>> let
>>> them be garbage collected. There will be no truncation of the last
>>> block(even if it is not full). This way , rather then garbage
>>> collecting
>>> all the blocks of a file , we will only be garbage collecting the
>>> remaining blocks.
>>>
>>> -vidur
>>>
>>>
>>> > HDFS assumes in hundreds of places that blocks never shrink. So,
>>> there
>>> is
>>> > no
>>> > option to truncate a block.
>>> >
>>> > -Todd
>>> >
>>> > On Mon, Jun 21, 2010 at 9:41 PM, Vidur Goyal
>>> > <vidur@students.iiit.ac.in>wrote:
>>> >
>>> >> Hi All,
>>> >>
>>> >> In FSNamesystem#startFileInternal , whenever there is a overwrite
>>> flag
>>> >> set
>>> >> , why is the INode removed from the namespace and a new
>>> >> INodeFileUnderConstruction is created. Why can't we use the convert
>>> the
>>> >> same INode to INodeFileUnderConstruction. And we start writing to
>>> the
>>> >> same
>>> >> blocks at the same datanodes (after incrementing the GS) followed by
>>> >> either truncating the remaining blocks(if the file size decreases)
>>> or
>>> >> allocating new blocks (if the file size increases). This will
>>> decrease
>>> >> data redundancy and the job of garbage collector and will increase
>>> >> security.
>>> >>
>>> >> vidur
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> This message has been scanned for viruses and
>>> >> dangerous content by MailScanner, and is
>>> >> believed to be clean.
>>> >>
>>> >>
>>> >
>>> >
>>> > --
>>> > Todd Lipcon
>>> > Software Engineer, Cloudera
>>> >
>>> > --
>>> > This message has been scanned for viruses and
>>> > dangerous content by MailScanner, and is
>>> > believed to be clean.
>>> >
>>> >
>>>
>>>
>>> --
>>> This message has been scanned for viruses and
>>> dangerous content by MailScanner, and is
>>> believed to be clean.
>>>
>>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>> --
>> This message has been scanned for viruses and
>> dangerous content by MailScanner, and is
>> believed to be clean.
>>
>>
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


Mime
View raw message