hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2379) 0.20: Allow block reports to proceed without holding FSDataset lock
Date Tue, 11 Oct 2011 19:29:12 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125314#comment-13125314

Todd Lipcon commented on HDFS-2379:

bq. In the new version of the patch, FSDatasetInterface.java changes are missing. Also asynchronous
scan thread change is missing as well. Want to make sure that it is intentional.

Not sure what you mean - I see them there.

bq. There are some lines that are more than 80 chars.

bq. Why do you want to deprecate #getBlockInfo()? If you have a valid reason, can you please
add information on the new method/mechanism that should be used instead of the deprecated
These methods were left only for the sake of the sanity-check code path. But given the below
comment, I've removed both the sanity check code path and the getBlockInfo method, since that's
the only spot it was used. (and it was private)

bq. SANITY_CHECK code can be removed.

bq. What happens to cases when volumeMap contains block but scanned block File does not exist
or scanned block file exists but volumeMap does not contain it?

The goal of this JIRA is to preserve the existing semantics - ie to produce an identical block
report as to what would have been produced if the whole scan had happened while under the
lock. So:

- If the block is in memory, but not on disk, the block is not reported. Note that we re-check
the existence on disk when we see this situation, to make sure it wasn't just that the block
was added after the scan. This code path handles the case where an administrator accidentally
rm -Rfs some blocks - we want to make sure they don't show up in the block report, so that
the NN can re-replicate.
- If the block is on disk, but not in memory, we do report it, but only after checking that
it's still there (with the lock held).
I've updated the comments in the code to clarify the above behaviors.

In the above cases, it might make some sense to actually update the in-memory map based on
what was seen on disk. But, that would change the semantics, which would be harder to verify.

bq. In the end, the scanned block info is made to look same as the in memory state. I am just
wondering, what is the need of the scan then?
The scan is made to look the same as the disk state. Anything places where we see a diff vs
memory, we then _recheck_ the disk for those blocks while holding the lock. So the semantics
should be the same as before.

Will upload another patch momentarily with the above fixes. I'll also run through a basic
manual test scenario of rm -Rfing some blocks and making sure they get re-replicated.
> 0.20: Allow block reports to proceed without holding FSDataset lock
> -------------------------------------------------------------------
>                 Key: HDFS-2379
>                 URL: https://issues.apache.org/jira/browse/HDFS-2379
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions:
>            Reporter: Todd Lipcon
>            Priority: Critical
>         Attachments: hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt
> As disks are getting larger and more plentiful, we're seeing DNs with multiple millions
of blocks on a single machine. When page cache space is tight, block reports can take multiple
minutes to generate. Currently, during the scanning of the data directories to generate a
report, the FSVolumeSet lock is held. This causes writes and reads to block, timeout, etc,
causing big problems especially for clients like HBase.
> This JIRA is to explore some of the ideas originally discussed in HADOOP-4584 for the
0.20.20x series.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message