hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-10597) Evaluate if we can have RPC client back off when server is under heavy load
Date Tue, 03 Jun 2014 03:27:03 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-10597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ming Ma updated HADOOP-10597:

    Attachment: HADOOP-10597.patch

Steve, thanks for the suggestions. Yes, it is useful to have server provide backoff parameters
to client.

Here is the initial patch to cover the basic idea. After we believe this is the right direction,
unit test and load tests will follow.

1. Enhance RetriableException for this purpose. When RPC queue is full, RetriableException
will be thrown back to the client with the backoff parameters in PB. The backoff parameters
are based on exponential back off.

2. Client/RetryPolicy can decide if and how to use the server hint. It is up to each retry
policy implementation. For NN HA RetryPolicy failoverOnNetworkException, it will use the server
hint when it is available. For NN non-HA scenario, HDFS-6478 needs to be fixed first. A new
policy called RetryUpToMaximumCountWithFixedSleepAndServerHint is provided as an example.

3. Backoff feature is turned off by default in RPC server.

> Evaluate if we can have RPC client back off when server is under heavy load
> ---------------------------------------------------------------------------
>                 Key: HADOOP-10597
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10597
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Ming Ma
>         Attachments: HADOOP-10597.patch
> Currently if an application hits NN too hard, RPC requests be in blocking state, assuming
OS connection doesn't run out. Alternatively RPC or NN can throw some well defined exception
back to the client based on certain policies when it is under heavy load; client will understand
such exception and do exponential back off, as another implementation of RetryInvocationHandler.

This message was sent by Atlassian JIRA

View raw message