hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-7960) The full block report should prune zombie storages even if they're not empty
Date Wed, 02 Sep 2015 02:03:46 GMT

     [ https://issues.apache.org/jira/browse/HDFS-7960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vinod Kumar Vavilapalli updated HDFS-7960:
------------------------------------------
    Fix Version/s: 2.6.1

[~sjlee0] backported this to 2.6.1, after fixing merge conflicts, java7 style log4j statements
and dropping changes from  TestBlockListAsLongs.java and TestTriggerBlockReport.java which
don't exist in 2.6.

I just pushed the commit to 2.6.1 after running compilation and TestBlockManager, TestNameNodePrunesMissingStorages,
TestBPOfferService, TestBlockHasMultipleReplicasOnSameDN, TestDataNodeVolumeFailure, TestDatanodeProtocolRetryPolicy,
TestDnRespectsBlockReportSplitThreshold, TestNNHandlesBlockReportPerStorage, TestNNHandlesCombinedBlockReport,
TestDeadDatanode and TestDNFencing which changed in the patch.

> The full block report should prune zombie storages even if they're not empty
> ----------------------------------------------------------------------------
>
>                 Key: HDFS-7960
>                 URL: https://issues.apache.org/jira/browse/HDFS-7960
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>            Reporter: Lei (Eddy) Xu
>            Assignee: Colin Patrick McCabe
>            Priority: Critical
>              Labels: 2.6.1-candidate
>             Fix For: 2.7.0, 2.6.1
>
>         Attachments: HDFS-7960-branch-2.6.1.txt, HDFS-7960.002.patch, HDFS-7960.003.patch,
HDFS-7960.004.patch, HDFS-7960.005.patch, HDFS-7960.006.patch, HDFS-7960.007.patch, HDFS-7960.008.patch
>
>
> The full block report should prune zombie storages even if they're not empty.  We have
seen cases in production where zombie storages have not been pruned subsequent to HDFS-7575.
 This could arise any time the NameNode thinks there is a block in some old storage which
is actually not there.  In this case, the block will not show up in the "new" storage (once
old is renamed to new) and the old storage will linger forever as a zombie, even with the
HDFS-7596 fix applied.  This also happens with datanode hotplug, when a drive is removed.
 In this case, an entire storage (volume) goes away but the blocks do not show up in another
storage on the same datanode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message