hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14660) wasb: improve throughput by 34% when account limit exceeded
Date Tue, 01 Aug 2017 21:24:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109805#comment-16109805

Hudson commented on HADOOP-14660:

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12093 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12093/])
HADOOP-14660 wasb: improve throughput by 34% when account limit (stevel: rev 778d4edd9adbe9519c3d6df65e45ddc8bb0ab2da)
* (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/ClientThrottlingAnalyzer.java
* (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/ClientThrottlingIntercept.java
* (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestClientThrottlingAnalyzer.java
* (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java
* (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestBlobOperationDescriptor.java
* (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlobOperationDescriptor.java
* (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AbstractWasbTestBase.java
* (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java

> wasb: improve throughput by 34% when account limit exceeded
> -----------------------------------------------------------
>                 Key: HADOOP-14660
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14660
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>            Reporter: Thomas Marquardt
>            Assignee: Thomas Marquardt
>         Attachments: HADOOP-14660-001.patch, HADOOP-14660-002.patch, HADOOP-14660-003.patch,
HADOOP-14660-004.patch, HADOOP-14660-005.patch, HADOOP-14660-006.patch, HADOOP-14660-007.patch,
HADOOP-14660-008.patch, HADOOP-14660-010.patch
> Big data workloads frequently exceed the Azure Storage max ingress and egress limits
(https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits).  For example,
the max ingress limit for a GRS account in the United States is currently 10 Gbps.  When the
limit is exceeded, the Azure Storage service fails a percentage of incoming requests, and
this causes the client to initiate the retry policy.  The retry policy delays requests by
sleeping, but the sleep duration is independent of the client throughput and account limit.
 This results in low throughput, due to the high number of failed requests and thrashing causes
by the retry policy.
> To fix this, we introduce a client-side throttle which minimizes failed requests and
maximizes throughput.  Tests have shown that this improves throughtput by ~34% when the storage
account max ingress and/or egress limits are exceeded. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message