hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7035) Make adding a new data directory to the DataNode an atomic operation and improve error handling
Date Wed, 27 Jan 2016 01:06:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118395#comment-15118395
] 

Arpit Agarwal commented on HDFS-7035:
-------------------------------------

Hi [~eddyxu], one question related to this change. {{DataNode#refreshVolumes}} sets DFS_DATANODE_DATA_DIR_KEY
to effectiveVolumes. This can be different from the input list passed in as a parameter as
it filters out failed volumes.
{code}
      conf.set(DFS_DATANODE_DATA_DIR_KEY,
          Joiner.on(",").join(effectiveVolumes));
      dataDirs = getStorageLocations(conf);
{code}

However this will conflict with the value set by {{ReconfigurableBase#reconfigureProperty}}
when {{DataNode#reconfigurePropertyImpl}} returns:
{code}
        reconfigurePropertyImpl(property, newVal);
        if (newVal != null) {
          getConf().set(property, newVal);
        } else {
          getConf().unset(property);
        }
{code}

Does this need fixing or am I understanding it wrong? Thanks.

> Make adding a new data directory to the DataNode an atomic operation and improve error
handling
> -----------------------------------------------------------------------------------------------
>
>                 Key: HDFS-7035
>                 URL: https://issues.apache.org/jira/browse/HDFS-7035
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: 2.5.0
>            Reporter: Lei (Eddy) Xu
>            Assignee: Lei (Eddy) Xu
>              Labels: 2.6.1-candidate
>             Fix For: 2.7.0, 2.6.1
>
>         Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch, HDFS-7035.001.combo.patch,
HDFS-7035.001.patch, HDFS-7035.002.patch, HDFS-7035.003.patch, HDFS-7035.003.patch, HDFS-7035.004.patch,
HDFS-7035.005.patch, HDFS-7035.007.patch, HDFS-7035.008.patch, HDFS-7035.009.patch, HDFS-7035.010.patch,
HDFS-7035.010.patch, HDFS-7035.011.patch, HDFS-7035.012.patch, HDFS-7035.013.patch, HDFS-7035.014.patch,
HDFS-7035.015.patch, HDFS-7035.016.patch
>
>
> It refactors {{DataStorage}} and {{BlockPoolSliceStorage}} to reduce the duplicate code
and supports atomic adding volume operations. Also it parallels loading data volume operation:
each thread loads one volume.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message