hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1783) Ability for HDFS client to write replicas in parallel
Date Fri, 25 May 2012 06:26:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283154#comment-13283154
] 

Lars Hofhansl commented on HDFS-1783:
-------------------------------------

So the proposal here is to have the client parallelize the requests to the replicas.

How about the option of having the first DN in the chain do that?
The client interacts with the DNs as before, but the first DN would parallelize to all (or
some) downstream DNs.
I.e. if there are 3 replicas the first DN would contact the 2nd and 3rd DN in parallel. Looking
at the PacketResponder code, that shouldn't be too hard to add.

Would be less parallel than doing it all in the client, but the client would not have to be
aware of the changes. On the flipside having this all in the client avoids DN changes.

As Dhruba points out, that would be quite useful for the HBase WALs, especially with HDFS-744.

                
> Ability for HDFS client to write replicas in parallel
> -----------------------------------------------------
>
>                 Key: HDFS-1783
>                 URL: https://issues.apache.org/jira/browse/HDFS-1783
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs client
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>
> The current implementation of HDFS pipelines the writes to the three replicas. This introduces
some latency for realtime latency sensitive applications. An alternate implementation that
allows the client to write all replicas in parallel gives much better response times to these
applications. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message