hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangmin Lee" <sangmin....@gmail.com>
Subject Re: HDFS : Question on INodeFileUnderConstruction targets filed ..
Date Mon, 04 Aug 2008 21:20:34 GMT
Dear dhruba,

Thanks you for your kind explanation.
Unfortunately, I failed to locate the code for use of 'targets' as you
described.
I was only able to see "targets" field be used by
dfs.FSEditLog.logOpenFile().
When LeaseMonitor finds a expired lease, it seems to get datanode locations
just from FSNamesystem.blockMaps rather than from the 'targets' field.
I guess I am missing something. Could you shed some more light on this?

I appreciate your help again.

Sangmin


On Fri, Aug 1, 2008 at 1:15 AM, Dhruba Borthakur <dhruba@gmail.com> wrote:

> The "targets" field is used to record the block location(s of the last
> allocated block fro a file. It is important to remember this at block
> allocation time itself so that data written to the block can be
> recovered if the client dies before it finished writing to the entire
> block.
>
> Take the case when the client started writing to the 3 datanodes in
> the pipeline for that block. Now, let's assume that the client died
> aftre writing 100 bytes to the first two replicas whereas the third
> replica has only 50 bytes in the block. The namenode holds a lease for
> this client. If the client dies prematurely, the namenode experiences
> a timeout for the client and triggers lease recovery for the file(s)
> that were being written by the client. The namenode has to ensure that
> all the three replicas are consistent with one another, otherwise
> readers of the block might see inconsistent data depending on which
> replica they access. Thus, the namenode has to remember which
> datanode(s) possibly has replicas of that block.
>
> hope this help,
> dhruba
>
> On Thu, Jul 31, 2008 at 12:27 PM, Sangmin Lee <sangmin.dev@gmail.com>
> wrote:
> > To clarify my question, what is the need for storing the locations of
> last
> > block without being reported by datanodes?
> >
> > On Thu, Jul 31, 2008 at 3:18 PM, Sangmin Lee <sangmin.dev@gmail.com>
> wrote:
> >
> >> Hi folks,
> >>
> >> I wonder why hadoop.dfs.INodeFileUnderConstruction Class has "targets"
> >> field.
> >> Can anyone explain this to me?
> >> any help would be appreciated.
> >>
> >> -sangmin
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message