hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9924) [umbrella] Nonblocking HDFS Access
Date Thu, 16 Jun 2016 21:56:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334770#comment-15334770

Kai Zheng commented on HDFS-9924:

Hi [~xiaobingo],

Thanks for the perf report! It looks great to me. I have some questions.
bq. In all cases, the larger number of calls tested, the less speedup. This could be because
larger number of calls saturated NameNode for longer time.
Could we verify this, maybe by trying different {{dfs.namenode.handler.count}}? 

Looks like you also mentioned {{dfs.permissions}} and {{dfs.datanode.handler.count}}, they're
not relevant to the perf test logic {{rename}} I guess, right?

How would you vision if we do the similar test for other operations like read/write?

By the way, currently we're trying to hack the codes and do a workload test in Hive, to see
if any real benefit as mentioned. Hope we could have some output sooner and provide some help
here for you guys.

> [umbrella] Nonblocking HDFS Access
> ----------------------------------
>                 Key: HDFS-9924
>                 URL: https://issues.apache.org/jira/browse/HDFS-9924
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Xiaobing Zhou
>         Attachments: Async-HDFS-Performance-Report.pdf, AsyncHdfs20160510.pdf
> This is an umbrella JIRA for supporting Nonblocking HDFS Access.
> Currently, all the API methods are blocking calls -- the caller is blocked until the
method returns.  It is very slow if a client makes a large number of independent calls in
a single thread since each call has to wait until the previous call is finished.  It is inefficient
if a client needs to create a large number of threads to invoke the calls.
> We propose adding a new API to support nonblocking calls, i.e. the caller is not blocked.
 The methods in the new API immediately return a Java Future object.  The return value can
be obtained by the usual Future.get() method.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message