Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <27630642.1192132910918.JavaMail.jira@brutus>
Date: Thu, 11 Oct 2007 13:01:50 -0700 (PDT)
From: "Sameer Paranjpye (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-2012) Periodic verification at the
 Datanode
In-Reply-To: <10538159.1191891590670.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534143 ] 

Sameer Paranjpye commented on HADOOP-2012:
------------------------------------------

> Even if we are writing 8 bytes, if there a problem during the write (hardware / driver error) ...

Maybe, but the number of writes that will be done as a result of this process is a couple of orders of magnitude less than those from regular block writes, replications, appends ... This is just noise.


> Periodic verification at the Datanode
> -------------------------------------
>
>                 Key: HADOOP-2012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2012
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>            Reporter: Raghu Angadi
>            Assignee: Raghu Angadi
>
> Currently on-disk data corruption on data blocks is detected only when it is read by the client or by another datanode.  These errors are detected much earlier if datanode can periodically verify the data checksums for the local blocks.
> Some of the issues to consider :
> - How should we check the blocks ( no more often than once every couple of weeks ?)
> - How do we keep track of when a block was last verfied ( there is a .meta file associcated with each lock ).
> - What action to take once a corruption is detected
> - Scanning should be done as a very low priority with rest of the datanode disk traffic in mind.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.