hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2012) Periodic verification at the Datanode
Date Wed, 10 Oct 2007 21:35:50 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533882

Raghu Angadi commented on HADOOP-2012:

Chatted with Rob and Dhruba. Here is the proposal :

- No data is stored persistently for now. If we need this we can add later.
- Datanode maintains last verification time for all the nodes in memory.
- When it first starts up, it keeps the blocks in random order. And then on wards verifies
them based on last verfication time. Initial random order is to avoid not verifying some blocks
across frequest restarts.
- Provides detailed stats on webui so that we can know if there any blocks that are not verified
in a log time. This also lets us know we should store any data persistenly.
- Instead of just deleting a block when a corruption is detected, it could let the namenode
decide whether to remove the block immediately or not. This might need a few changes on Namenode.

> Periodic verification at the Datanode
> -------------------------------------
>                 Key: HADOOP-2012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2012
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>            Reporter: Raghu Angadi
>            Assignee: Raghu Angadi
> Currently on-disk data corruption on data blocks is detected only when it is read by
the client or by another datanode.  These errors are detected much earlier if datanode can
periodically verify the data checksums for the local blocks.
> Some of the issues to consider :
> - How should we check the blocks ( no more often than once every couple of weeks ?)
> - How do we keep track of when a block was last verfied ( there is a .meta file associcated
with each lock ).
> - What action to take once a corruption is detected
> - Scanning should be done as a very low priority with rest of the datanode disk traffic
in mind.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message