hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-5092) Add support for incremental cache reports
Date Wed, 22 Jan 2014 21:56:53 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Wang updated HDFS-5092:
------------------------------

    Labels: caching  (was: )

> Add support for incremental cache reports
> -----------------------------------------
>
>                 Key: HDFS-5092
>                 URL: https://issues.apache.org/jira/browse/HDFS-5092
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode, namenode
>            Reporter: Colin Patrick McCabe
>            Assignee: Andrew Wang
>            Priority: Minor
>              Labels: caching
>
> The initial {{cacheReport}} patch at HDFS-5051 does frequent full reports of DN cache
state. Better would be a scheme similar to how block reports are currently done: send incremental
cache reports on every heartbeat (seconds), and full reports on a longer time scale (minutes
to hours). This should reduce network traffic and allow us to make incremental reports even
faster.
> As per discussion on HDFS-5051, we should also roll-up the following review comments:
> - Remove gen stamp and length from {{cacheReport}}, unnecessary until we do auto-caching
of appended data
> - Only jitter full cache reports, similar to how full block reports are jittered
> - On DN startup, skip all cache reports until the cache is populated. The NN can just
assume the DN cache is empty in the meantime.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message