hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-8782) Upgrade to block ID-based DN storage layout delays DN registration
Date Wed, 15 Jul 2015 16:44:04 GMT
Haohui Mai created HDFS-8782:
--------------------------------

             Summary: Upgrade to block ID-based DN storage layout delays DN registration
                 Key: HDFS-8782
                 URL: https://issues.apache.org/jira/browse/HDFS-8782
             Project: Hadoop HDFS
          Issue Type: Bug
            Reporter: Haohui Mai
            Priority: Critical


We have seen multiple incidents at production sites that there are long delays for DNs to
register to the NN when upgrading to post 2.6 release.

Further investigation shows that the DN is blocked when upgrading the storage layout introduced
in HDFS-6482. The new storage layout requires making up to 64k directories in the underlying
file system. Unfortunately the current implementation calls {{mkdirs()}} sequentially and
upgrades each volume in sequential order.

As a result, upgrading a DN with a lot of disks or with blocks that have random block ID takes
a long time (usually in hours), and the DN won't register to the NN unless it finishes upgrading
all the storage directory. The excessive delays confuse operations and break the assumption
of rolling upgrades.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message