hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4258) Rename of Being Written Files
Date Mon, 03 Dec 2012 01:27:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508432#comment-13508432
] 

Tsz Wo (Nicholas), SZE commented on HDFS-4258:
----------------------------------------------

One way to solve this problem is to add INode ID.  We could add INode ID without increasing
memory usage since both modification time and access time are currently 8 bytes, which supports
{noformat}
2^64 / (1000 * 3600 * 24 * 365.25) = 584542046 years,
{noformat}
 which is way too much.  If only 5 bytes is used, it supports
{noformat}
2^40/(1000 * 3600 * 24 * 365.25) = 34.84 years.
{noformat}
It may already good enough.  We may also change the accuracy from milliseconds to hundredth
or tenth seconds.  Then, 5 bytes supports 348.4 years and 3484 years respectively.  The INode
ID is stored in the remaining 6 bytes, which supports 2^48 = 281 trillion of INodes. We may
play around with the parameters more and have the following table.
|| atime/mtime || Time period supported (with milliseconds accuracy) || INode ID || \# INodes
supported |
| 40 bits | 34.84 years | 48 bits | 281.47 trillions |
| 41 bits | 69.68 years | 46 bits |  70.37 trillions |
| 42 bits | 139.37 years | 44 bits |  17.59 trillions |
| 43 bits | 278.73 years | 42 bits |  4.40 trillions |
| 44 bits | 557.46 years | 40 bits |  1.10 trillions |


                
> Rename of Being Written Files
> -----------------------------
>
>                 Key: HDFS-4258
>                 URL: https://issues.apache.org/jira/browse/HDFS-4258
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client, namenode
>            Reporter: Tsz Wo (Nicholas), SZE
>
> When a being written file or it's ancestor directories is renamed, the path in the file
lease is also renamed.  Then the writer of the file usually will fail since the file path
in the writer is not updated.
> Moreover, I think there is a bug as follow:
> # Client writes 0's to F_0="/foo/file" and writes 1's to F_1="/bar/file" at the same
time.
> # Rename /bar to /baz
> # Rename /foo to /bar
> Then, writing to F_0 will fail since /foo/file does not exist anymore but writing to
F_1 may succeed since /bar/file exits as a different file.  In such case, the content of /bar/file
could be partly 0's and partly 1's.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message