hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-158) dfs should allocate a random blockid range to a file, then assign ids sequentially to blocks in the file
Date Tue, 30 May 2006 21:37:30 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-158?page=comments#action_12413911 ] 

Doug Cutting commented on HADOOP-158:

I would think that random allocation would make collisions more likely, not less.  We always
know which block ids are used by complete files.  The concern is only about block ids which
have been recently allocated to a file, but the file is somehow not yet complete.  So, with
sequential allocation, a collision can only happen if the probe key (the next block id to
allocate) wraps all the way around before a file is completed, while with random allocation
it can happen much more frequently.  We simply have to make sure that probe key increments
are logged to the edits file along with other file system changes.  Am I missing something?

> dfs should allocate a random blockid range to a file, then assign ids sequentially to
blocks in the file
> --------------------------------------------------------------------------------------------------------
>          Key: HADOOP-158
>          URL: http://issues.apache.org/jira/browse/HADOOP-158
>      Project: Hadoop
>         Type: Bug

>   Components: dfs
>     Versions: 0.1.0
>     Reporter: Doug Cutting
>     Assignee: Konstantin Shvachko
>      Fix For: 0.4

> A random number generator is used to allocate block ids in dfs.  Sometimes a block id
is allocated that is already used in the filesystem, which causes filesystem corruption.
> A short-term fix for this is to simply check when allocating block ids whether any file
is already using the newly allocated id, and, if it is, generate another one.  There can still
be collisions in some rare conditions, but these are harder to fix and will wait, since this
simple fix will handle the vast majority of collisions.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

View raw message