hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10597) Evaluate if we can have RPC client back off when server is under heavy load
Date Wed, 15 Oct 2014 04:32:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14171966#comment-14171966
] 

Ming Ma commented on HADOOP-10597:
----------------------------------

Thanks, Chris.

1. The first two experiments belong to "user A is doing some bad things, measure user B's
read latency.". The rest of the experiments are done under single user to measure the performance
implication under different loads.
2. We can use client backoff without FCQ. But it is less interesting, given it could penalize
good clients. That is because in the current implementation, the criteria RPC server uses
to decide if it needs to ask client to back off is whether the RPC call queue is full. We
can improve this criteria later if this isn't enough.
3. The experiment results are based on "client driven retry interval" policy. It means the
server only asks the client to back off; RPC client will decide retry policy. In NN HA setup,
that will be FailoverOnNetworkExceptionRetry which does exponential back off.
 

> Evaluate if we can have RPC client back off when server is under heavy load
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-10597
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10597
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Ming Ma
>            Assignee: Ming Ma
>         Attachments: HADOOP-10597-2.patch, HADOOP-10597.patch, MoreRPCClientBackoffEvaluation.pdf,
RPCClientBackoffDesignAndEvaluation.pdf
>
>
> Currently if an application hits NN too hard, RPC requests be in blocking state, assuming
OS connection doesn't run out. Alternatively RPC or NN can throw some well defined exception
back to the client based on certain policies when it is under heavy load; client will understand
such exception and do exponential back off, as another implementation of RetryInvocationHandler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message