hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dhruba Borthakur" <dhr...@gmail.com>
Subject Re: HDFS : Question on INodeFileUnderConstruction targets filed ..
Date Fri, 01 Aug 2008 06:15:34 GMT
The "targets" field is used to record the block location(s of the last
allocated block fro a file. It is important to remember this at block
allocation time itself so that data written to the block can be
recovered if the client dies before it finished writing to the entire
block.

Take the case when the client started writing to the 3 datanodes in
the pipeline for that block. Now, let's assume that the client died
aftre writing 100 bytes to the first two replicas whereas the third
replica has only 50 bytes in the block. The namenode holds a lease for
this client. If the client dies prematurely, the namenode experiences
a timeout for the client and triggers lease recovery for the file(s)
that were being written by the client. The namenode has to ensure that
all the three replicas are consistent with one another, otherwise
readers of the block might see inconsistent data depending on which
replica they access. Thus, the namenode has to remember which
datanode(s) possibly has replicas of that block.

hope this help,
dhruba

On Thu, Jul 31, 2008 at 12:27 PM, Sangmin Lee <sangmin.dev@gmail.com> wrote:
> To clarify my question, what is the need for storing the locations of last
> block without being reported by datanodes?
>
> On Thu, Jul 31, 2008 at 3:18 PM, Sangmin Lee <sangmin.dev@gmail.com> wrote:
>
>> Hi folks,
>>
>> I wonder why hadoop.dfs.INodeFileUnderConstruction Class has "targets"
>> field.
>> Can anyone explain this to me?
>> any help would be appreciated.
>>
>> -sangmin
>>
>

Mime
View raw message