hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gera Shegalov (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-8182) Implement topology-aware CDN-style caching
Date Mon, 20 Apr 2015 03:11:58 GMT
Gera Shegalov created HDFS-8182:

             Summary: Implement topology-aware CDN-style caching
                 Key: HDFS-8182
                 URL: https://issues.apache.org/jira/browse/HDFS-8182
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: hdfs-client, namenode
    Affects Versions: 2.6.0
            Reporter: Gera Shegalov

To scale reads of hot blocks in large clusters, it would be beneficial if we could read a
block across the ToR switches only once. Example scenarios are localization of binaries, MR
distributed cache files for map-side joins and similar. There are multiple layers where this
could be implemented (YARN service or individual apps such as MR) but I believe it is best
done in HDFS or even common FileSystem to support as many use cases as possible. 

The life cycle could look like this e.g. for the YARN localization scenario:
1. inputStream = fs.open(path, ..., CACHE_IN_RACK)
2. instead of reading from a remote DN directly, NN tells the client to read via the local
DN1 and the DN1 creates a replica of each block.

When the next localizer on DN2 in the same rack starts it will learn from NN about the replica
in DN1 and the client will read from DN1 using the conventional path.

When the application ends the AM or NM's can instruct the NN in a fadvise DONTNEED style,
it can start telling DN's to discard extraneous replica.

This message was sent by Atlassian JIRA

View raw message