hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Duo Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9924) [umbrella] Nonblocking HDFS Access
Date Wed, 25 Jan 2017 09:17:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837416#comment-15837416
] 

Duo Zhang commented on HDFS-9924:
---------------------------------

Any updates here? Seems no commit on HDFS-9924 branch for a long time...

I can help a bit here as I still want to move the FanOutOneBlockAsyncDFSOutputStream into
HDFS rather than maintain it in HBase...

I think the problem here is that, our interface is blocking. It is really awkward to implement
async stuffs on top of an blocking interface so I do not like the current approach. I think
we can either

1. Use grpc instead of the current rpc. Add a port unification service in front of the grpc
server and the old rpc server to support both grpc client and old client. Yeah we need to
write lots of code if we choose this way, but I think most code are just boilerplate. Another
benefit is that multi language support will be much easier if we use standard grpc.

2. Use grpc but do not use the HTTP/2 transport, implement our own transport. I haven't tried
this yet but grpc-java does support customized transport so I think it is possible. The benefit
is that we do not need port unification service at server side and do not need to maintain
two implementations at server side.

3. Use the old protobuf rpc interface and implement a new rpc framework. The benefit is that
we also do not need port unification service at server side and do not need to maintain two
implementations at server side. And one more thing is that we do not need to upgrade protobuf
to 3.x.

4. As said in the design doc above, generate new interfaces which return a CompletableFuture
based on the old blocking interface. And add a new feature in the current rpc implementation
to support the new interface.

I'm OK with any of the approach above.  Can start working on branch HDFS-9924 after we decide
which one to use.

Thanks.

> [umbrella] Nonblocking HDFS Access
> ----------------------------------
>
>                 Key: HDFS-9924
>                 URL: https://issues.apache.org/jira/browse/HDFS-9924
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Xiaobing Zhou
>         Attachments: AsyncHdfs20160510.pdf, Async-HDFS-Performance-Report.pdf
>
>
> This is an umbrella JIRA for supporting Nonblocking HDFS Access.
> Currently, all the API methods are blocking calls -- the caller is blocked until the
method returns.  It is very slow if a client makes a large number of independent calls in
a single thread since each call has to wait until the previous call is finished.  It is inefficient
if a client needs to create a large number of threads to invoke the calls.
> We propose adding a new API to support nonblocking calls, i.e. the caller is not blocked.
 The methods in the new API immediately return a Java Future object.  The return value can
be obtained by the usual Future.get() method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message