hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2012) Periodic verification at the Datanode
Date Tue, 30 Oct 2007 19:16:50 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12538891

Raghu Angadi commented on HADOOP-2012:

> The metadata about the entire Datanode is stored in the VERSION file. It is possible
that we can store the last verified-blockid in this file (instead of adding a new file).

VERSION file contains only the basic information needed for starting up a Datanode and is
vital for Datanode startup. It is not updated at runtime. I don't think it is suited for for

Regd continuous scanning, I think most users would not prefer that. Even 5MB per sec is close
to 20% of a single disk read and much more if the read/write is not very sequential (happens
when there are multiple reads and writes).

We can certainly make the SCAN_PERIOD and throttle bandwidth configurable (may be not in hadoop-defaults.xml),
where power users can tweak it as appropriate. I know there is strong resistance for adding
any config variables :). But I personal opinion is that some more config's that 99% of users
don't need to worry about because of good defaults are ok. 

> Periodic verification at the Datanode
> -------------------------------------
>                 Key: HADOOP-2012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2012
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>            Reporter: Raghu Angadi
>            Assignee: Raghu Angadi
>             Fix For: 0.16.0
>         Attachments: HADOOP-2012.patch, HADOOP-2012.patch, HADOOP-2012.patch, HADOOP-2012.patch
> Currently on-disk data corruption on data blocks is detected only when it is read by
the client or by another datanode.  These errors are detected much earlier if datanode can
periodically verify the data checksums for the local blocks.
> Some of the issues to consider :
> - How should we check the blocks ( no more often than once every couple of weeks ?)
> - How do we keep track of when a block was last verfied ( there is a .meta file associcated
with each lock ).
> - What action to take once a corruption is detected
> - Scanning should be done as a very low priority with rest of the datanode disk traffic
in mind.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message