hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-171) need standard API to set dfs replication = high
Date Thu, 27 Apr 2006 02:47:03 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-171?page=comments#action_12376622 ] 

Doug Cutting commented on HADOOP-171:

>  We can implement short highReplicationHint()

That would work, but it doesn't seem like the simplest API.  But if folks feel strongly I
can live with it.

> Then copyFromLocalFile() would use that value to create files.

I don't think we want copyFromLocal to always do that.  I think we'd end up with something


That's okay.  If the filesystem grows or shrinks a lot then it might become inadequate.  But
whatever, if most folks prefer this, I'd be willing to go along.

Perhaps we need to distingush in the API between the requested replication count and the actual
replication count.  Then one can set the requested level to Small.MAX_VALUE and query the
actual value to see how the fs interprets this.

> need standard API to set dfs replication = high
> -----------------------------------------------
>          Key: HADOOP-171
>          URL: http://issues.apache.org/jira/browse/HADOOP-171
>      Project: Hadoop
>         Type: New Feature

>   Components: dfs
>     Versions: 0.2
>     Reporter: Doug Cutting
>     Assignee: Konstantin Shvachko

> There should be a standard way to indicate that files should be highly replicated, appropriate
for files that all nodes will read.  This should be settable both on file creation and for
already-existing files.  Perhaps specifying a particular replication value, like Short.MAX_VALUE,
or zero, can be used to signal this.  The level should not be constant, but should be relative
to the cluster size and network topography.  If more nodes are added or if nodes are deleted,
the actual replication count should increase or decrease.
> Initially, all that is needed is an API to specify this.  It could initially be implemented
with a constant (e.g., 10) or with something related to the number of datanodes (sqrt?), and
needn't auto-adjust as the cluster size changes  That is only  the long-term goal.
> When JobClient copies job files (job.xml & job.jar) into the job's filesystem, it
should specify this replication level.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

View raw message