hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Kumar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9666) Enable hdfs-client to read even remote SSD/RAM prior to local disk replica to improve random read
Date Fri, 06 Apr 2018 04:54:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16427968#comment-16427968

Ajay Kumar commented on HDFS-9666:

[~yangjiandan] thanks for updating the patch. Overall it looks good. Few comments:
 * DFSInputStream
 ** Personally i think we should merge getBestNodeDNAddrPair and getBestNodeDNAddrPairRemoteSsdFirst
as most of the functionality is same.
 ** Improve chooseDataNode javadoc for remoteSsdFirst (L860) to something like "if true read
remote SSD/RAM replica first if local Disks are HDD"
 **  Improve javadoc for getBestNodeDNAddrPairRemoteSsdFirst to mention the selection strategy.
"Read from local node if a) If block is on SSD. b) If no other replica exist on SSD or RAM
 Read from remote node if local node replica is on HDD and remote node replica is on SSD/RAM"
 * TestDFSInputStream#testReadSsdFirstWithSsd
 ** Typo in L222  & L259. We should include expected storageId in Assert text. i.e 
"Should be storageID3." & "Should be storageID1." respectively.
 ** Also we should assert expected storage type in both tests.

> Enable hdfs-client to read even remote SSD/RAM prior to local disk replica to improve
random read
> -------------------------------------------------------------------------------------------------
>                 Key: HDFS-9666
>                 URL: https://issues.apache.org/jira/browse/HDFS-9666
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 2.6.0, 2.7.0
>            Reporter: ade
>            Assignee: Jiandan Yang 
>            Priority: Major
>         Attachments: HDFS-9666.0.patch, HDFS-9666.001.patch, HDFS-9666.002.patch, HDFS-9666.003.patch,
> We want to improve random read performance of HDFS for HBase, so enabled the heterogeneous
storage in our cluster. But there are only ~50% of datanode & regionserver hosts with
SSD. we can set hfile with only ONE_SSD not ALL_SSD storagepolicy and the regionserver on
none-SSD host can only read the local disk replica . So we developed this feature in hdfs
client to read even remote SSD/RAM prior to local disk replica.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message