hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12082) BlockInvalidateLimit value is incorrectly set after namenode heartbeat interval reconfigured
Date Tue, 04 Jul 2017 14:40:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073760#comment-16073760
] 

Weiwei Yang commented on HDFS-12082:
------------------------------------

Proposed a patch to fix this issue. Otherwise if user reconfigures namenode interval, the
value of property {{dfs.block.invalidate.limit}} will be always overwritten.

The fix simply honors the configuration of {{dfs.block.invalidate.limit}} and use it for block
invalidate limit, and this will not change when heartbeat interval changes. Reason is, following
formula doesn't really work

{code}
// Default heartbeat is 3s, unless heartbeat is set to bigger than 50s,
// it is always 1000
(1) final int blockInvalidateLimit = Math.max(20*(int)(heartbeatIntervalSeconds),
          DFSConfigKeys.DFS_BLOCK_INVALIDATE_LIMIT_DEFAULT);

// We will not reach the default value here, because we always load defaults
// from hdfs-default.xml. If the property is not found in hdfs-site.xml,
// it simply returns the default value from hdfs-default.xml, which is 1000.
// Even blockInvalidateLimit is something else, it doesn't count.
(2) this.blockInvalidateLimit = conf.getInt(
          DFSConfigKeys.DFS_BLOCK_INVALIDATE_LIMIT_KEY, blockInvalidateLimit);
{code}

so right now there are two cases
# {{dfs.block.invalidate.limit}} not set explicitly, blockInvalidateLimit=1000
# {{dfs.block.invalidate.limit}} is explicitly set, blockInvalidateLimit=<value_of_the_property>

in this case, why we still need (1) ? I think we can remove it.

> BlockInvalidateLimit value is incorrectly set after namenode heartbeat interval reconfigured

> ---------------------------------------------------------------------------------------------
>
>                 Key: HDFS-12082
>                 URL: https://issues.apache.org/jira/browse/HDFS-12082
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs, namenode
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>         Attachments: HDFS-12082.001.patch
>
>
> HDFS-1477 provides an option to reconfigured namenode heartbeat interval without restarting
the namenode. When the heartbeat interval is reconfigured, {{blockInvalidateLimit}} gets recounted
> {code}
>  this.blockInvalidateLimit = Math.max(20 * (int) (intervalSeconds),
>         DFSConfigKeys.DFS_BLOCK_INVALIDATE_LIMIT_DEFAULT);
> {code}
> this doesn't honor the existing value set by {{dfs.block.invalidate.limit}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message