hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-2936) Provide a better way to specify a HDFS-wide minimum replication requirement
Date Wed, 18 Apr 2012 20:30:41 GMT

     [ https://issues.apache.org/jira/browse/HDFS-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Robert Joseph Evans updated HDFS-2936:
--------------------------------------

    Target Version/s: 2.0.0, 3.0.0  (was: 0.23.2, 0.24.0)
    
> Provide a better way to specify a HDFS-wide minimum replication requirement
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-2936
>                 URL: https://issues.apache.org/jira/browse/HDFS-2936
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.23.0
>            Reporter: Harsh J
>            Assignee: Harsh J
>
> Currently, if an admin would like to enforce a replication factor for all files on his
HDFS, he does not have a way. He may arguably set dfs.replication.min but that is a very hard
guarantee and if the pipeline can't afford that number for some reason/failure, the close()
does not succeed on the file being written and leads to several issues.
> After discussing with Todd, we feel it would make sense to introduce a second config
(which is ${dfs.replication.min} by default) which would act as a minimum specified replication
for files. This is different than dfs.replication.min which also ensures that many replicas
are recorded before completeFile() returns... perhaps something like ${dfs.replication.min.user}.
We can leave dfs.replication.min alone for hard-guarantees and add ${dfs.replication.min.for.block.completion}
which could be left at 1 even if dfs.replication.min is >1, and let files complete normally
but not be of a low replication factor (so can be monitored and accounted-for later).
> I'm prefering the second option myself. Will post a patch with tests soon.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message