hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmed Mahran (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-8416) Short circuit remote reads from shared storage
Date Sat, 16 May 2015 13:37:59 GMT
Ahmed Mahran created HDFS-8416:

             Summary: Short circuit remote reads from shared storage
                 Key: HDFS-8416
                 URL: https://issues.apache.org/jira/browse/HDFS-8416
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: datanode, hdfs-client, nfs, performance
            Reporter: Ahmed Mahran

In a Hadoop cluster configuration that employs a shared storage system, HDFS read and write
operations are very expensive in terms of network bandwidth consumption.

For a DFS client to read a block from a remote datanode, the block is transmitted first from
the shared storage to the datanode then from the datanode to the DFS client. Short circuiting
the shared storage to datanode hop and allowing the client to directly access the shared storage
would improve the performance substantially.

This document describes the issue and provides a hack for the remote read.

This message was sent by Atlassian JIRA

View raw message