hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-12502) nntop should support a category based on FilesInGetListingOps
Date Tue, 24 Oct 2017 22:00:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217777#comment-16217777
] 

Zhe Zhang edited comment on HDFS-12502 at 10/24/17 9:59 PM:
------------------------------------------------------------

For some reason we were getting over 600k~700k FilesInGetListing per second during a few days,
causing spikes in GC time. Single op processing time (inside the FSNLock, measured via {{FSNReadLockOpNameNanosAvgTime}})
increased by over 50%. And we don't have any tool find the abusing workload. Yes we are using
fair call queue but similar to NNTop it only considers number of ops; and each large listing
is 100 times as expensive as a getFileInfo. We should probably also extend fair call queue
to consider the cost of each op.

I'll work on reverting the patch now.


was (Author: zhz):
For some reason we were getting over 600k~700k FilesInGetListing per second during a few days,
causing spikes in GC time. Single op processing time (inside the FSNLock, measured via {{FSNReadLockOpNameNanosAvgTime}})
increased by over 50%. And we don't have any tool find the abusing workload. Yes we are using
fair call queue but similar to NNTop it only considers number of ops; and each large listing
is 100 times as expensive as a getFileInfo. We should probably also extend fair call queue
to consider the cost of each op.

> nntop should support a category based on FilesInGetListingOps
> -------------------------------------------------------------
>
>                 Key: HDFS-12502
>                 URL: https://issues.apache.org/jira/browse/HDFS-12502
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: metrics
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>             Fix For: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0
>
>         Attachments: HDFS-12502.00.patch, HDFS-12502.01.patch, HDFS-12502.02.patch, HDFS-12502.03.patch,
HDFS-12502.04.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode slowness. The aggregate
cost of listing ops is proportional to the {{FilesInGetListingOps}} rather than the number
of listing ops. Therefore it'd be very useful for nntop to support this category.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message