hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Rutherglen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2004) Enable replicating and pinning files to a data node
Date Sat, 28 May 2011 21:20:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040676#comment-13040676
] 

Jason Rutherglen commented on HDFS-2004:
----------------------------------------

bq. To be clear, "pinning" isn't really what we want here. We want to request that the NN
make a local replica – similar to what the balancer does.

Right, however pinning would be great and I think it's achievable using a modified placement
policy and Balancer that skips moving files from a location based on a pattern.  This would
effectively 'pin' blocks to a DataNode.

bq. All client software should work with non-local replicas, but if it knows it's going to
need a copy for a while, pulling one over makes some sense.

Unfortunately Lucene cannot work with non-local replicas.  For example a single query with
4 clauses would open 4 input streams because each could perform a seek.

bq. Yes, we could have HBase use some local storage to cache blocks, but then we're faced
with a potentially large increase in storage requirements.

I think that's OK, eg, it's more of a configuration/operations problem/option.  For some HBase
clusters, the current sometimes non-local replicas is optimal, for others it may not be.


> Enable replicating and pinning files to a data node
> ---------------------------------------------------
>
>                 Key: HDFS-2004
>                 URL: https://issues.apache.org/jira/browse/HDFS-2004
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: balancer
>    Affects Versions: 0.23.0
>            Reporter: Jason Rutherglen
>
> Some HDFS applications require that a given file is on the local DataNode.  The functionality
created here will allow pinning the file to any DataNode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message