hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9821) HDFS configuration should accept friendly time units
Date Wed, 24 Feb 2016 21:02:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163793#comment-15163793
] 

Colin Patrick McCabe commented on HDFS-9821:
--------------------------------------------

It seems odd to me to have a configuration key that explicitly says "millis" and configure
it in terms of hours (or some other time unit).  On the other hand,  if we have to introduce
new unit-less keys for every affected key, that's quite a few new keys.

I did a quick survey and found all these:
{code}
dfs.client.write.byte-array-manager.count-reset-time-period-ms
dfs.client.file-block-storage-locations.timeout.millis
dfs.client.socketcache.expiryMsec
dfs.client.write.exclude.nodes.cache.expiry.interval.millis
dfs.datanode.lazywriter.interval.sec
dfs.namenode.storageinfo.defragment.interval.ms
dfs.namenode.storageinfo.defragment.timeout.ms
dfs.namenode.edit.log.autoroll.check.interval.ms
dfs.namenode.lazypersist.file.scrub.interval.sec
dfs.content-summary.sleep-microsec
dfs.datanode.oob.timeout-ms
dfs.datanode.cache.revocation.timeout.ms
dfs.datanode.cache.revocation.polling.ms
dfs.namenode.path.based.cache.refresh.interval.ms
dfs.namenode.startup.delay.block.deletion.sec
dfs.datanode.scan.period.hours
dfs.namenode.path.based.cache.retry.interval.ms
dfs.blockreport.intervalMsec
dfs.namenode.full.block.report.lease.length.ms
dfs.cachereport.intervalMsec
dfs.client.read.shortcircuit.streams.cache.expiry.ms
dfs.client.mmap.cache.timeout.ms
dfs.client.mmap.retry.timeout.ms
dfs.client.short.circuit.replica.stale.threshold.ms
...
{code}

at that point I stopped counting since there were too many.  So yes, I guess re-using the
existing key names is more pragmatic.

> HDFS configuration should accept friendly time units
> ----------------------------------------------------
>
>                 Key: HDFS-9821
>                 URL: https://issues.apache.org/jira/browse/HDFS-9821
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode, namenode
>    Affects Versions: 2.8.0
>            Reporter: Arpit Agarwal
>            Assignee: Xiaobing Zhou
>
> HDFS configuration keys that define time intervals use units inconsistently (Hours, seconds,
milliseconds).
> Not all keys have the unit as part of their name. Related keys may use different units
e.g. {{dfs.blockreport.intervalMsec}} accepts msec while {{dfs.blockreport.initialDelay}}
accepts seconds. Milliseconds is rarely useful as a time unit which makes these values hard
to parse when reading config files.
> We can either
> # Let existing keys use friendly units e.g. 100ms, 60s, 5m, 1d, 6w etc. This can be done
compatibly since there will be no conflict with existing valid configuration. If no suffix
is specified just default to the current time unit.
> # Just deprecate the existing keys and define new ones that accept friendly units.
> We continue to use fine-grained time units (usually ms) internally in code and also accept
"ms" option for tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message