hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Rutherglen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2004) Enable replicating and pinning files to a data node
Date Sat, 28 May 2011 21:20:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040676#comment-13040676

Jason Rutherglen commented on HDFS-2004:

bq. To be clear, "pinning" isn't really what we want here. We want to request that the NN
make a local replica – similar to what the balancer does.

Right, however pinning would be great and I think it's achievable using a modified placement
policy and Balancer that skips moving files from a location based on a pattern.  This would
effectively 'pin' blocks to a DataNode.

bq. All client software should work with non-local replicas, but if it knows it's going to
need a copy for a while, pulling one over makes some sense.

Unfortunately Lucene cannot work with non-local replicas.  For example a single query with
4 clauses would open 4 input streams because each could perform a seek.

bq. Yes, we could have HBase use some local storage to cache blocks, but then we're faced
with a potentially large increase in storage requirements.

I think that's OK, eg, it's more of a configuration/operations problem/option.  For some HBase
clusters, the current sometimes non-local replicas is optimal, for others it may not be.

> Enable replicating and pinning files to a data node
> ---------------------------------------------------
>                 Key: HDFS-2004
>                 URL: https://issues.apache.org/jira/browse/HDFS-2004
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: balancer
>    Affects Versions: 0.23.0
>            Reporter: Jason Rutherglen
> Some HDFS applications require that a given file is on the local DataNode.  The functionality
created here will allow pinning the file to any DataNode.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message