hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5051) Propagate cache status information from the DataNode to the NameNode
Date Wed, 14 Aug 2013 20:49:48 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740166#comment-13740166

Andrew Wang commented on HDFS-5051:

I included the gen stamp and length in the {{cacheReport}} to handle caching newly appended
data. I guess the gen stamp is unnecessary, but the DN isn't going to automatically mlock
newly appended data, so the NN needs to somehow realize that the cached length is shorter
than the new length and ask the DN to recache at the new length. Alternatively, I guess the
DN could automatically mlock appended data, but there are quota implications there.

On startup, I agree that we can skip cache reports until the cache is populated. I also agree
that jittering doesn't matter as much if it's ticking on such a short time scale. I guess
I could have cleaned this up rather than just changing the default cache report period like
Colin asked.

However, since we want to eventually have both incremental and full reports, let's just ape
how block reports work; don't jitter the incremental reports, but do jitter the start time
for the full reports and afterwards tick at a regular interval. Let's clean up all these issues
in the incremental cache report JIRA (HDFS-5092); if this sounds good, I'll edit the JIRA
description with these todo items.
> Propagate cache status information from the DataNode to the NameNode
> --------------------------------------------------------------------
>                 Key: HDFS-5051
>                 URL: https://issues.apache.org/jira/browse/HDFS-5051
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>            Reporter: Colin Patrick McCabe
>            Assignee: Andrew Wang
>         Attachments: hdfs-5051-1.patch, hdfs-5051-2.patch
> The DataNode needs to inform the NameNode of its current cache state. Let's wire up the
RPCs and stub out the relevant methods on the DN and NN side.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message