hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Duo Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7966) New Data Transfer Protocol via HTTP/2
Date Mon, 03 Aug 2015 22:45:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14652698#comment-14652698

Duo Zhang commented on HDFS-7966:

I do not have enough machines to test the scenario... What I see if I create lots of thread
to read from datanode concurrently is that HTTP/2 will start the request almost at the same
time, but TCP will start the request one by one(maybe tens by tens where the number is cpu
count). So there won't be a situation that DN really handle lots of concurrent read from client,
and the context switch maybe small than HTTP/2 implementation since we also have a ThreadPool
besides EventLoopGroup in HTTP/2 connection. And what make things worse is that our client
is not event driven so we can not reduce the thread count of client...
Let me see if I can make a scenario that HTTP/2 fast than TCP...

> New Data Transfer Protocol via HTTP/2
> -------------------------------------
>                 Key: HDFS-7966
>                 URL: https://issues.apache.org/jira/browse/HDFS-7966
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Haohui Mai
>            Assignee: Qianqian Shi
>              Labels: gsoc, gsoc2015, mentor
>         Attachments: GSoC2015_Proposal.pdf, TestHttp2LargeReadPerformance.svg, TestHttp2Performance.svg,
> The current Data Transfer Protocol (DTP) implements a rich set of features that span
across multiple layers, including:
> * Connection pooling and authentication (session layer)
> * Encryption (presentation layer)
> * Data writing pipeline (application layer)
> All these features are HDFS-specific and defined by implementation. As a result it requires
non-trivial amount of work to implement HDFS clients and servers.
> This jira explores to delegate the responsibilities of the session and presentation layers
to the HTTP/2 protocol. Particularly, HTTP/2 handles connection multiplexing, QoS, authentication
and encryption, reducing the scope of DTP to the application layer only. By leveraging the
existing HTTP/2 library, it should simplify the implementation of both HDFS clients and servers.

This message was sent by Atlassian JIRA

View raw message