hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls
Date Wed, 25 May 2016 23:21:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15301094#comment-15301094

stack commented on HADOOP-12910:

The patch here converts one method only? Is the intent to do all methods (along w/ the spec
suggested by [~steve_l]?)

Is this issue for an AsyncFileSystem or for an async rename only? Are we targeting H3 only
or is there some thought that this could get pulled back into H2?

bq. Futures are a good match for the use case where the consumer wants to kick of a multitude
of async requests and wait until they are all done to make progress, but we've found that
there are also compelling use cases where you want a small amount of logic and further async
I/O in a completion handler, so I might recommend supporting both Future-based results as
well as callback-based results.

A few of us (mainly [~Apache9]), are looking at being able to go async against hdfs. There
is already a stripped down async subset of dfsclient that we are using to write our WALs done
by [~Apache9] that uses way less resources while going much faster (see HBASE-14790). As Duo
says, we want to push this up into HDFS, and given our good experience with this effort, we
want to convert over more of our HDFS connection to be async. Parking a resource waiting on
a Future to complete or keeping some list/queue of Futures which we check on a period to see
if it is 'done' is much less attractive (and less performant) to our being instead notified
on completion -- a callback (as [~bobhansen] suggests above in the comment repeated here).
 Ideally we'd like to move our interaction with HDFS to be event-driven (ultimately backing
this up all the ways into the guts of the regionserver, but that is another story)

OK if we put up suggested patch that say presumes jdk8/h3 only and instead of returning Future,
returns jdk8 CompletableFuture? Chatting yesterday, we think we could consume/feed HDFS in
a non-blocking way if we got back a CompletableFuture (or we could add a callback handler
as a param on a method if folks preferred that?). We'd put up a sketch patch, and if amenable,
we could start up a sympathetic spec doc as a subtask so code and spec arrive at the same


> Add new FileSystem API to support asynchronous method calls
> -----------------------------------------------------------
>                 Key: HADOOP-12910
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12910
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Xiaobing Zhou
>         Attachments: HADOOP-12910-HDFS-9924.000.patch, HADOOP-12910-HDFS-9924.001.patch,
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a better
name).  All the APIs in FutureFileSystem are the same as FileSystem except that the return
type is wrapped by Future, e.g.
> {code}
>   //FileSystem
>   public boolean rename(Path src, Path dst) throws IOException;
>   //FutureFileSystem
>   public Future<Boolean> rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message