hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yiqun Lin (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-14313) Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du
Date Tue, 23 Jul 2019 15:27:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891132#comment-16891132
] 

Yiqun Lin edited comment on HDFS-14313 at 7/23/19 3:26 PM:
-----------------------------------------------------------

Hi [~leosun08],
1.
{quote}
I don't use the component specific impl class in the common module. to update in GetSpaceUsed
of common model is just to make subclasses inheritable. And that to update in CommonConfigurationKeys
of common model is to print threshold time,which should be moved to DFSConfigKeys and is
more appropriate..
{quote}
I don't think it's necessary to have two new config to make the threshold time configurable,
the hard-coded way should be enough. I mean we can define a hard-coded threadold time value
like 1000ms in ReplicaCachingGetSpaceUsed. So that we don't need to do any change in Common.

2. Can you add comment for the potential dead lock issue in the method deepCopyReplica? That
will let others know the context. 

Besides above two points, others make sense to me.



was (Author: linyiqun):
Hi [~leosun08],
1.
{quote}
I don't use the component specific impl class in the common module. to update in GetSpaceUsed
of common model is just to make subclasses inheritable. And that to update in CommonConfigurationKeys
of common model is to print threshold time,which should be moved to DFSConfigKeys and is
more appropriate..
{quote}
I don't think it's necessary to have two new config to make the threshold time configurable,
the hard-coded way should be enough. I mean we can define a hard-coded threadold time value
like 1000ms in ReplicaCachingGetSpaceUsed. So that we don't need to do any change in Common.

2. Can you add comment for the potential dead lock issue in the method deepCopyReplica? That
will be let others know the context. 

Besides above two points, others make sense to me.


> Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory  instead of df/du
> ----------------------------------------------------------------------------------------
>
>                 Key: HDFS-14313
>                 URL: https://issues.apache.org/jira/browse/HDFS-14313
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode, performance
>    Affects Versions: 2.6.0, 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.0
>            Reporter: Lisheng Sun
>            Assignee: Lisheng Sun
>            Priority: Major
>         Attachments: HDFS-14313.000.patch, HDFS-14313.001.patch, HDFS-14313.002.patch,
HDFS-14313.003.patch, HDFS-14313.004.patch, HDFS-14313.005.patch, HDFS-14313.006.patch, HDFS-14313.007.patch
>
>
> There are two ways of DU/DF getting used space that are insufficient.
>  #  Running DU across lots of disks is very expensive and running all of the processes
at the same time creates a noticeable IO spike.
>  #  Running DF is inaccurate when the disk sharing by multiple datanode or other servers.
>  Getting hdfs used space from  FsDatasetImpl#volumeMap#ReplicaInfos in memory is very
small and accurate. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message