hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7966) New Data Transfer Protocol via HTTP/2
Date Mon, 03 Aug 2015 21:43:08 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14652579#comment-14652579
] 

Haohui Mai commented on HDFS-7966:
----------------------------------

bq. What's the upside of this new implementation? 

Performance is definitely one important factor. One of the motivation is to improve the efficiency
of DN when there are hundreds of thousands of reads by reducing the overhead of context switches.
[~Apache9], do you have any performance numbers on this scenario?

HTTP/2-based DTP also serves as a building block of the next-level of innovation, just to
quote the description in the jira:

{quote}
This jira explores to delegate the responsibilities of the session and presentation layers
to the HTTP/2 protocol. Particularly, HTTP/2 handles connection multiplexing, QoS, authentication
and encryption, reducing the scope of DTP to the application layer only. By leveraging the
existing HTTP/2 library, it should simplify the implementation of both HDFS clients and servers.
{quote}

bq. If it were the same performance but had other redeeming qualities (e.g. less code) then
it's still worth consideration.

This is designed to be a new code path so that it is compatible with older releases. You can
still rely on the old DTP protocol depending on the application scenario.

> New Data Transfer Protocol via HTTP/2
> -------------------------------------
>
>                 Key: HDFS-7966
>                 URL: https://issues.apache.org/jira/browse/HDFS-7966
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Haohui Mai
>            Assignee: Qianqian Shi
>              Labels: gsoc, gsoc2015, mentor
>         Attachments: GSoC2015_Proposal.pdf, TestHttp2LargeReadPerformance.svg, TestHttp2Performance.svg,
TestHttp2ReadBlockInsideEventLoop.svg
>
>
> The current Data Transfer Protocol (DTP) implements a rich set of features that span
across multiple layers, including:
> * Connection pooling and authentication (session layer)
> * Encryption (presentation layer)
> * Data writing pipeline (application layer)
> All these features are HDFS-specific and defined by implementation. As a result it requires
non-trivial amount of work to implement HDFS clients and servers.
> This jira explores to delegate the responsibilities of the session and presentation layers
to the HTTP/2 protocol. Particularly, HTTP/2 handles connection multiplexing, QoS, authentication
and encryption, reducing the scope of DTP to the application layer only. By leveraging the
existing HTTP/2 library, it should simplify the implementation of both HDFS clients and servers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message