hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Danny Becker (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-14069) Better debuggability for datanode decommissioning
Date Mon, 12 Nov 2018 23:16:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Danny Becker updated HDFS-14069:
--------------------------------
    Description: 
Currently, we only provide "minLiveReplicas" per DN that is being decommissioned, this is
not enough info because it is difficult to determine which blocks are on their last replica.
We have two design options:
 # Add it to the existing report, on top of minLiveReplicas
 ** Advantages:
 *** Initial debugging information would be more thorough
 *** Easier initial implementation
 ** Disadvantages:
 *** Add load to normal NN operation by checking every time a DN is decommissioned
 *** More difficult to add debugging information later on
 # Create a new api for querying more detailed info about one DN
 ** Advantages:
 *** We wouldnt be adding more load to the NN in normal operation
 *** Much easier to extend in the future with more info
 ** Disadvantages:
 *** Getting the info on demand for this case will be much more expensive actually, cause
we will have to find all the blocks on that DN, and then go through all the blocks again and
count how many replicas we have etc.

  was:
Currently, we only provide "minLiveReplicas" per DN that is being decommissioned, this is
not enough info because it is difficult to determine which blocks are on their last replica.
We have two design options:
 # Add it to the existing report, on top of minLiveReplicas
 ** Advantages:
 *** Initial debugging information would be more thorough
 ** Disadvantages:
 *** Add load to normal NN operation by checking every time a DN is decommissioned
 # Create a new api for querying more detailed info about one DN
 ** Advantages:
 *** We wouldnt be adding more load to the NN in normal operation
 *** Much easier to extend in the future with more info
 ** Disadvantages:
 *** Getting the info on demand for this case will be much more expensive actually, cause
we will have to find all the blocks on that DN, and then go through all the blocks again and
count how many replicas we have etc.


> Better debuggability for datanode decommissioning
> -------------------------------------------------
>
>                 Key: HDFS-14069
>                 URL: https://issues.apache.org/jira/browse/HDFS-14069
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode, hdfs, namenode
>            Reporter: Danny Becker
>            Priority: Major
>         Attachments: HDFS-14069.000.patch
>
>
> Currently, we only provide "minLiveReplicas" per DN that is being decommissioned, this
is not enough info because it is difficult to determine which blocks are on their last replica.
We have two design options:
>  # Add it to the existing report, on top of minLiveReplicas
>  ** Advantages:
>  *** Initial debugging information would be more thorough
>  *** Easier initial implementation
>  ** Disadvantages:
>  *** Add load to normal NN operation by checking every time a DN is decommissioned
>  *** More difficult to add debugging information later on
>  # Create a new api for querying more detailed info about one DN
>  ** Advantages:
>  *** We wouldnt be adding more load to the NN in normal operation
>  *** Much easier to extend in the future with more info
>  ** Disadvantages:
>  *** Getting the info on demand for this case will be much more expensive actually, cause
we will have to find all the blocks on that DN, and then go through all the blocks again and
count how many replicas we have etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message