hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10475) Adding metrics for long FSNamesystem read and write locks
Date Tue, 13 Sep 2016 21:43:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488523#comment-15488523
] 

Zhe Zhang commented on HDFS-10475:
----------------------------------

Thanks Erik for the analysis.

bq. Move the logging/metrics logic into FSNamesystemLock
Maybe we should also consider reusing and extending {{InstrumentedLock}}. 

bq. Add new methods, (read|write)Unlock(operation),
Good idea. Since a thread doesn't carry the name of the calling method, I don't think we can
avoid adding new unlock APIs.

As you stated, the most tricky part is updating a centralized metrics object in parallel.
From the above analysis and our offline discussion, I see several options:
# We can have a regular (not synchronized) {{HashMap}} with pre-populated keys. Each key maps
to a "suspicious" {{FSNamesystem}} op, such as {{listStatus}}. It's the simplest and most
lightweight (low memory and time overhead) option. The main drawback is having to hand-pick
(and later maintain) a set of ops. Assuming that we have a clear idea of the set of "suspicious"
ops and the set is both stable over time and consistent across different deployments, this
would be the best option.
# On the other extreme, we can have a {{ConcurrentHashMap}} and accurately add each op names.
This is also simple, but is the most heavyweight (both memory and time). From memory perspective,
I think there are ~100 ops using read/write locks so it's not too much of a concern. On the
other hand, updating a single {{ConcurrentHashMap}}, although outside the fsn lock, could
still be expensive. If we pursue this option we should use an appropriate level of thread-local
aggregation / throttling before "flushing" to the centralized map.

Pinging [~chris.douglas] and [~vagarychen] since this is related to HDFS-10742. Also pinging
[~andrew.wang]: does an Op->LockTime map look useful in your production cases?

> Adding metrics for long FSNamesystem read and write locks
> ---------------------------------------------------------
>
>                 Key: HDFS-10475
>                 URL: https://issues.apache.org/jira/browse/HDFS-10475
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Xiaoyu Yao
>            Assignee: Erik Krogen
>
> This is a follow up of the comment on HADOOP-12916 and [here|https://issues.apache.org/jira/browse/HDFS-9924?focusedCommentId=15310837&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15310837]
add more metrics and WARN/DEBUG logs for long FSD/FSN locking operations on namenode similar
to what we have for slow write/network WARN/metrics on datanode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message