hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sameer Paranjpye (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-158) dfs should allocate a random blockid range to a file, then assign ids sequentially to blocks in the file
Date Tue, 30 May 2006 20:04:30 GMT
     [ http://issues.apache.org/jira/browse/HADOOP-158?page=all ]

Sameer Paranjpye updated HADOOP-158:
------------------------------------

        Summary: dfs should allocate a random blockid range to a file, then assign ids sequentially
to blocks in the file  (was: dfs block ids sometimes collide)
    Fix Version: 0.4
                     (was: 0.3)
        Version: 0.1.0

Block id collisions have largely beed addressed by the fix in HADOOP-146. The namenode first
checks for the presence of a randomly generated blockid before assigning it to a block.

Longer term, we should implement a scheme where the namenode allocates large blockid ranges
to files. When a file is created the namenode generates a 5-byte integer, a range-id, for
the file, checking for collisions and re-generating the range-id if necessary. Blocks for
the new file are then assigned 8-byte block ids sequentially, where the high 5-bytes are the
range-id and the low 3-bytes correspond to the block number within the file. Blocks in a file
then get the ids rangeid-0, rangeid-1, ..., rangeid-(N-1), where N is the number of blocks
in the file. This lets us assign upto a trillion ranges and lets each file grow to 0.5, 1,
2 ... petabytes depending on whether the block size is 32, 64, 128 ... MB. The scheme has
the additional benefit of saving some memory per block at the namenode.

There is the scenario of a file being deleted while a node holding some of its blocks is down,
having the files range-id re-assigned to another file and seeing collisions when the node
later comes back. To get around this, the namenode tags each block in a file with the creation
time of the file. When a collision occurs the timestamps will be different, the most recent
timestamp wins and old blocks are discarded.



> dfs should allocate a random blockid range to a file, then assign ids sequentially to
blocks in the file
> --------------------------------------------------------------------------------------------------------
>
>          Key: HADOOP-158
>          URL: http://issues.apache.org/jira/browse/HADOOP-158
>      Project: Hadoop
>         Type: Bug

>   Components: dfs
>     Versions: 0.1.0
>     Reporter: Doug Cutting
>     Assignee: Konstantin Shvachko
>      Fix For: 0.4

>
> A random number generator is used to allocate block ids in dfs.  Sometimes a block id
is allocated that is already used in the filesystem, which causes filesystem corruption.
> A short-term fix for this is to simply check when allocating block ids whether any file
is already using the newly allocated id, and, if it is, generate another one.  There can still
be collisions in some rare conditions, but these are harder to fix and will wait, since this
simple fix will handle the vast majority of collisions.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message