hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "KenCao (Jira)" <j...@apache.org>
Subject [jira] [Created] (HDFS-14809) Make a new BlockReader for hdfs client lib
Date Sun, 01 Sep 2019 03:58:00 GMT
KenCao created HDFS-14809:

             Summary: Make a new BlockReader  for hdfs client lib
                 Key: HDFS-14809
                 URL: https://issues.apache.org/jira/browse/HDFS-14809
             Project: Hadoop HDFS
          Issue Type: Bug
    Affects Versions: 2.6.0
            Reporter: KenCao

as we known, the hdfs client java lib uses BlockReaderLocal for short circuit read by default,
which allocate shared memory first, and make a slot within it. After all these steps, it will
request the fds from the DataNode. However, the slot and shared memory sturcture is only used
by DataNode when uncaching replicas, the client process can work well just with the fds asked
later and it is nearly impossible to cache replicas in product environment. The api to release
fds is called by client only with the slot given, the fds is close in the client process finally. 

so i think we can make a new BlockReader implementation which just requests the fds, and it
will reduce the rpc calls from 3(allocate shm, request fds, release fds) to 1(request fds).

This message was sent by Atlassian Jira

To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

View raw message