hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yoram Arnon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-158) dfs should allocate a random blockid range to a file, then assign ids sequentially to blocks in the file
Date Tue, 30 May 2006 21:30:30 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-158?page=comments#action_12413910 ] 

Yoram Arnon commented on HADOOP-158:
------------------------------------

It could, but that would make upgrading an existing file system harder: one would have to
"compress" the id's before upgrading, then keep track of the high-water-mark file id. The
upgrade wouldn't be as smooth as when selecting 'clean' ranges randomly, and would require
metadata conversion. Blocks on individual datanodes may even need to change their ids.
OR, one could copy/move the entire filesystem to a new, clean one.
The selected method allows for a seamless upgrade, requiring no conversion, and would work
equally well with a 2^64 id address space.

> dfs should allocate a random blockid range to a file, then assign ids sequentially to
blocks in the file
> --------------------------------------------------------------------------------------------------------
>
>          Key: HADOOP-158
>          URL: http://issues.apache.org/jira/browse/HADOOP-158
>      Project: Hadoop
>         Type: Bug

>   Components: dfs
>     Versions: 0.1.0
>     Reporter: Doug Cutting
>     Assignee: Konstantin Shvachko
>      Fix For: 0.4

>
> A random number generator is used to allocate block ids in dfs.  Sometimes a block id
is allocated that is already used in the filesystem, which causes filesystem corruption.
> A short-term fix for this is to simply check when allocating block ids whether any file
is already using the newly allocated id, and, if it is, generate another one.  There can still
be collisions in some rare conditions, but these are harder to fix and will wait, since this
simple fix will handle the vast majority of collisions.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message