hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9924) [umbrella] Asynchronous HDFS Access
Date Fri, 27 May 2016 20:23:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304703#comment-15304703

Daryn Sharp commented on HDFS-9924:

I agree that we can't protect the NN with a "slow" client.  If hive wasn't mentioned, I'd
probably have not thought to be concerned.  Hive's unnecessary heavy write op load has already
placed it on my naughty list internally.  This well-intentioned feature is enabling hive to
place much more stress on the NN.

I glanced at HADOOP-12916 but I don't think it's sufficient, actually I think it will have
the opposite effect - hopefully I misinterpreted the design.  Priority level is based on incoming
call rate.  The client is told to backoff if the NN is being "too slow" for that priority.
 The problem I see is:
* User1 has a write op heavy load
* User2 perhaps a wide job is doing 20X the ops of User1, but is read heavy
* User2 is dropped into a lower priority bucket based on call rate
* User1 drives up the avg processing time with costly writes
* User2 rapidly forced to backoff primarily from an accounting flaw of every write op is cumulatively
billed to all subsequent queued calls (processing time includes lock wait from prior calls),
artificially driving up the average
* User2 retries and is double billed on call rate since user is charged even if forced to
* User2 falls into even a lower bucket, enabling User1 to further drive up processing time
* Finally... User3 & User4 were running long getContentSummary calls - which acquires/releases
lock.  The prioritized writes made those summary calls appear to take a min.  Explodes average.
 Users are told to backoff from mostly empty queues.

> [umbrella] Asynchronous HDFS Access
> -----------------------------------
>                 Key: HDFS-9924
>                 URL: https://issues.apache.org/jira/browse/HDFS-9924
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Xiaobing Zhou
>         Attachments: AsyncHdfs20160510.pdf
> This is an umbrella JIRA for supporting Asynchronous HDFS Access.
> Currently, all the API methods are blocking calls -- the caller is blocked until the
method returns.  It is very slow if a client makes a large number of independent calls in
a single thread since each call has to wait until the previous call is finished.  It is inefficient
if a client needs to create a large number of threads to invoke the calls.
> We propose adding a new API to support asynchronous calls, i.e. the caller is not blocked.
 The methods in the new API immediately return a Java Future object.  The return value can
be obtained by the usual Future.get() method.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message