hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joe Pallas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7722) DataNode#checkDiskError should also remove Storage when error is found.
Date Sun, 29 Mar 2015 23:59:52 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386029#comment-14386029

Joe Pallas commented on HDFS-7722:

Sure, [~eddyxu].  HDFS-5194 is about improving support for alternative storage implementations.
 Assuming that volumes always correspond to directories in the local file system limits the
ability to implement other storage architectures, such as a directly attached object store
or perhaps a block-level device with a lightweight user-level "file system" layer optimized
for storing block replicas.

The {{FsDatasetSpi}} interface tries to abstract out the essentials of storing replicas, and
{{FsVolumeSpi}} is an abstract unit of storage used by the dataset to represent some subset
of all the available storage (typically a single drive in the default implementation).  Advertising
that volumes are directories doesn't just limit alternative implementations, it also makes
it harder to evolve the default implementation, because the scope of changes is harder to
determine once implementation details leak through the abstraction.

That's my perspective.  Maintaining these abstractions takes some some work, but it has benefits
for readability/maintainability of the default implementation as well as for alternative implementations.

> DataNode#checkDiskError should also remove Storage when error is found.
> -----------------------------------------------------------------------
>                 Key: HDFS-7722
>                 URL: https://issues.apache.org/jira/browse/HDFS-7722
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.6.0
>            Reporter: Lei (Eddy) Xu
>            Assignee: Lei (Eddy) Xu
>             Fix For: 2.7.0
>         Attachments: HDFS-7722.000.patch, HDFS-7722.001.patch, HDFS-7722.002.patch, HDFS-7722.003.patch,
> When {{DataNode#checkDiskError}} found disk errors, it removes all block metadatas from
{{FsDatasetImpl}}. However, it does not removed the corresponding {{DataStorage}} and {{BlockPoolSliceStorage}}.

> The result is that, we could not directly run {{reconfig}} to hot swap the failure disks
without changing the configure file.

This message was sent by Atlassian JIRA

View raw message