Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 13 Sep 2016 21:43:20 +0000 (UTC)
From: "Zhe Zhang (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12974901.1464806292000.564540.1473803000714@Atlassian.JIRA>
In-Reply-To: <JIRA.12974901.1464806292000@Atlassian.JIRA>
References: <JIRA.12974901.1464806292000@Atlassian.JIRA> <JIRA.12974901.1464806292633@arcas>
Subject: [jira] [Commented] (HDFS-10475) Adding metrics for long
 FSNamesystem read and write locks
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 13 Sep 2016 21:43:22 -0000


    [ https://issues.apache.org/jira/browse/HDFS-10475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488523#comment-15488523 ] 

Zhe Zhang commented on HDFS-10475:
----------------------------------

Thanks Erik for the analysis.

bq. Move the logging/metrics logic into FSNamesystemLock
Maybe we should also consider reusing and extending {{InstrumentedLock}}. 

bq. Add new methods, (read|write)Unlock(operation),
Good idea. Since a thread doesn't carry the name of the calling method, I don't think we can avoid adding new unlock APIs.

As you stated, the most tricky part is updating a centralized metrics object in parallel. From the above analysis and our offline discussion, I see several options:
# We can have a regular (not synchronized) {{HashMap}} with pre-populated keys. Each key maps to a "suspicious" {{FSNamesystem}} op, such as {{listStatus}}. It's the simplest and most lightweight (low memory and time overhead) option. The main drawback is having to hand-pick (and later maintain) a set of ops. Assuming that we have a clear idea of the set of "suspicious" ops and the set is both stable over time and consistent across different deployments, this would be the best option.
# On the other extreme, we can have a {{ConcurrentHashMap}} and accurately add each op names. This is also simple, but is the most heavyweight (both memory and time). From memory perspective, I think there are ~100 ops using read/write locks so it's not too much of a concern. On the other hand, updating a single {{ConcurrentHashMap}}, although outside the fsn lock, could still be expensive. If we pursue this option we should use an appropriate level of thread-local aggregation / throttling before "flushing" to the centralized map.

Pinging [~chris.douglas] and [~vagarychen] since this is related to HDFS-10742. Also pinging [~andrew.wang]: does an Op->LockTime map look useful in your production cases?

> Adding metrics for long FSNamesystem read and write locks
> ---------------------------------------------------------
>
>                 Key: HDFS-10475
>                 URL: https://issues.apache.org/jira/browse/HDFS-10475
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Xiaoyu Yao
>            Assignee: Erik Krogen
>
> This is a follow up of the comment on HADOOP-12916 and [here|https://issues.apache.org/jira/browse/HDFS-9924?focusedCommentId=15310837&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15310837] add more metrics and WARN/DEBUG logs for long FSD/FSN locking operations on namenode similar to what we have for slow write/network WARN/metrics on datanode.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org