hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rushabh S Shah (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-7928) Scanning blocks from disk during rolling upgrade startup takes a lot of time if disks are busy
Date Wed, 18 Mar 2015 15:31:39 GMT

     [ https://issues.apache.org/jira/browse/HDFS-7928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rushabh S Shah updated HDFS-7928:
---------------------------------
    Attachment: HDFS-7928-v1.patch

Addressed all the daryn's comment except for the 8th one.
I assume he made that comment considering the layout version change in future.
This will make the readReplicasFromFile function blow up and throw an exception.
In case of any exception while reading the cache file, this code will fall back to read from
the disk.
Please review and comment.

> Scanning blocks from disk during rolling upgrade startup takes a lot of time if disks
are busy
> ----------------------------------------------------------------------------------------------
>
>                 Key: HDFS-7928
>                 URL: https://issues.apache.org/jira/browse/HDFS-7928
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.6.0
>            Reporter: Rushabh S Shah
>            Assignee: Rushabh S Shah
>         Attachments: HDFS-7928-v1.patch, HDFS-7928.patch
>
>
> We observed this issue in rolling upgrade to 2.6.x on one of our cluster.
> One of the disks was very busy and it took long time to scan that disk compared to other
disks.
> Seeing the sar (System Activity Reporter) data we saw that the particular disk was very
busy performing IO operations.
> Requesting for an improvement during datanode rolling upgrade.
> During shutdown, we can persist the whole volume map on the disk and let the datanode
read that file and create the volume map during startup  after rolling upgrade.
> This will not require the datanode process to scan all the disk and read the block.
> This will significantly improve the datanode startup time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message