hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmytro Molkov (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-854) Datanode should scan devices in parallel to generate block report
Date Tue, 30 Mar 2010 00:29:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851177#action_12851177

Dmytro Molkov commented on HDFS-854:

The errors in the build system again look like they are not related to the actual patch. Two
errors in BlockScanner are

java.lang.RuntimeException: java.util.zip.ZipException: error reading zip file
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1715)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1529)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1475)
	at org.apache.hadoop.conf.Configuration.set(Configuration.java:596)
	at org.apache.hadoop.conf.Configuration.setLong(Configuration.java:731)
	at org.apache.hadoop.hdfs.TestDatanodeBlockScanner.blockCorruptionRecoveryPolicy(TestDatanodeBlockScanner.java:277)
	at org.apache.hadoop.hdfs.TestDatanodeBlockScanner.testBlockCorruptionRecoveryPolicy(TestDatanodeBlockScanner.java:269)

and there is one more error with HDFSProxy which cannot have anything to do with my code changes.

> Datanode should scan devices in parallel to generate block report
> -----------------------------------------------------------------
>                 Key: HDFS-854
>                 URL: https://issues.apache.org/jira/browse/HDFS-854
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.22.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.22.0
>         Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1
> A Datanode should scan its disk devices in parallel so that the time to generate a block
report is reduced. This will reduce the startup time of a cluster.
> A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K
blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate
the first block report.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message