hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei (Eddy) Xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6877) Avoid calling checkDisk and stop active BlockReceiver thread when an HDFS volume is removed during a write.
Date Wed, 24 Sep 2014 03:38:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145835#comment-14145835
] 

Lei (Eddy) Xu commented on HDFS-6877:
-------------------------------------

[Sorry for reposting. Mistakenly clicked "Add" during editing comments]

[~cmccabe] Thank you very much for your quick reviews.

I anticipated a behavior that the application is writing the block in very slow speed. For
instance, a light traffic service opens a log file and only appends 10KB logs after waking
up for every 30 seconds? Or some applications that dump immediate results (e.g., 1MB) after
every 10 minutes? In such cases, the BlockReceiver thread can live for a long time, until
the block is full and then be closed, even its {{ReplicaInfo}} has been removed from {{FsDatasetImpl}}.
But from the user point of view, after hdfs dfsadmin -reconfig status indicates the reconfigure
task has finished, the user might expect to be able to remove the disks physically anytime?
Since BlockReceiver thread is still alive for uncertain time and it opens file descriptor
on the disk, it might be difficult to explain to user why the disk can not be umount and how
long they should wait for it.

{code}
    for (ReplicaInPipeline pip : activeWriters) {
      try {
        LOG.info("Stopping active writer.");
        pip.stopWriter(datanode.getDnConf().getXceiverStopTimeout());
      } catch (IOException e) {
        LOG.warn("IOException when stopping active writer.", e);
      }
    }
{code}

The reasons that I moved the above out of synchronized section are
# {{stopWrite()}} is calling interrupt() and join(), which are slow. So I attempted to make
the synchronized section in removeVolumes(), which blocks access to {{FsDatasetImpl}},  shorter.
# After this synchronized critical section, all ReplicaInfo of active BlockReceivers have
been removed. Thus these BlockReceiver threads are outstanding threads. Since the {{stopWrite()}}
is slow, the thread might either be stopped by {{stopWrite()}} or reach the {{finalizeBlock()}}
before there is a chance to interrupt the thread. But either way, the thread can exit and
tells upstream sender that the block being written is bad.

It might be better to let these two behaviors (e.g., interruption writers and catching ReplicaNotFoundException
for {{finalizeBlock()}}) return same result. What do you think?

> Avoid calling checkDisk and stop active BlockReceiver thread when an HDFS volume is removed
during a write.
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-6877
>                 URL: https://issues.apache.org/jira/browse/HDFS-6877
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: 2.5.0
>            Reporter: Lei (Eddy) Xu
>            Assignee: Lei (Eddy) Xu
>         Attachments: HDFS-6877.000.consolidate.txt, HDFS-6877.000.delta-HDFS-6727.txt,
HDFS-6877.001.combo.txt, HDFS-6877.001.patch, HDFS-6877.002.patch, HDFS-6877.003.patch, HDFS-6877.004.patch,
HDFS-6877.005.patch
>
>
> Avoid calling checkDisk and stop active BlockReceiver thread when an HDFS volume is removed
during a write.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message