hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei-Chiu Chuang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9923) Datanode disk failure handling should be improved (consistently?)
Date Fri, 18 Mar 2016 19:15:33 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201988#comment-15201988

Wei-Chiu Chuang commented on HDFS-9923:

HADOOP-8640 is another incidence of the same issue.
As Todd found in HADOOP-8640,
The current behavior of DU which propagates the error to callers is not a good idea, because
the callers have no way of knowing how to handle it.

What about logging the error, but don't propagate the error to the caller. Instead, initiate
block scanner right away?

> Datanode disk failure handling should be improved (consistently?)
> -----------------------------------------------------------------
>                 Key: HDFS-9923
>                 URL: https://issues.apache.org/jira/browse/HDFS-9923
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>            Reporter: Wei-Chiu Chuang
>              Labels: failure-handling, supportability
> Disk failures are hard to handle. This JIRA is created to discuss/improve disk failure
handling in a better/consistent manner.
> For one thing, disks can fail in multiple different ways: the hardware might be failing,
disk space is full, checksum error ... For others, hardware abstracts out the details, so
it's hard for software to handle them.
> There are currently three disk check mechanisms in HDFS, as far as I know: {{BlockScanner}},
{{BlockPoolSlice#checkDirs}} and {{DU}}. Disk errors are handled differently. 
> This JIRA is more focused on {{DU}} error handling. {{DU}} may emit errors like this:
> {noformat}
> 2016-02-18 02:23:36,224 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
Caught exception while scanning /data/8/dfs/dn/current.
> Will throw later.
> ExitCodeException exitCode=1: du: cannot access `/data/8/dfs/dn/current/BP-1018136951-
> 1088686909': Input/output error
> du: cannot access `/data/8/dfs/dn/current/BP-1018136951-':
> ut/output error
> {noformat}
> I found {{DU}} errors are not handled consistently while working on HDFS-9908 (Datanode
should tolerate disk scan failure during NN handshake), and it all depends on who catches
the exception.
> For example, 
> * if DU returns error during NN handshake, DN will not be able to join the cluster at
> * however, if the same exception is caught in {{BlockPoolSlice#saveDfsUsed}}, data node
will only log a warning and do nothing (HDFS-5498).
> * in some cases, the exception handler invokes {{BlockPoolSlice#checkDirs}}, but since
it only checks three directories, it is very unlikely to find the files that have the error.
> So my ask is: should the error be handled in a consistent manner? Should data node report
to the name nodes about the disk failures (this is the BlockScanner approach), and should
data node takes this volume offline automatically if DU returns an error? (this is the {{checkDirs}}

This message was sent by Atlassian JIRA

View raw message