Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Date: Thu, 6 Aug 2015 21:28:06 +0000 (UTC)
From: "Rushabh S Shah (JIRA)" <jira@apache.org>
To: hdfs-dev@hadoop.apache.org
Message-ID: <JIRA.12852708.1438896438000.395280.1438896486055@Atlassian.JIRA>
In-Reply-To: <JIRA.12852708.1438896438000@Atlassian.JIRA>
References: <JIRA.12852708.1438896438000@Atlassian.JIRA>
 <JIRA.12852708.1438896438757@arcas>
Subject: [jira] [Created] (HDFS-8869) Don't mark storages as failed before
 first block report
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Rushabh S Shah created HDFS-8869:
------------------------------------

             Summary: Don't mark storages as failed before first block report
                 Key: HDFS-8869
                 URL: https://issues.apache.org/jira/browse/HDFS-8869
             Project: Hadoop HDFS
          Issue Type: Bug
    Affects Versions: 2.7.0
            Reporter: Rushabh S Shah
            Assignee: Daryn Sharp


Creating this ticket on behalf of [~daryn].

Heartbeat processing performs the failed storage check. The DN reports its storages and any prior missing storages, ex. unique storage id upgrade, are marked failed. The heartbeat monitor removes all blocks associated to the failed storage. A replication storm ensues for all blocks on the node.

Eventually the DN block reports for the new storages - up to 15m later for large clusters. Now the NN has many excess blocks to invalidate. If the cluster has failed over in the past 24h, ex. rolling upgrade, the standby gone active will queue the block invalidations which triggers the severe performance degradation of HDFS-8674 which has been greatly lessened but is still an issue.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)