Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 16 Aug 2017 12:51:00 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13087073.1500001198000.43096.1502887860545@Atlassian.JIRA>
In-Reply-To: <JIRA.13087073.1500001198000@Atlassian.JIRA>
References: <JIRA.13087073.1500001198000@Atlassian.JIRA> <JIRA.13087073.1500001198882@jira-lw-us.apache.org>
Subject: [jira] [Updated] (HADOOP-14660) wasb: improve throughput by 34%
 when account limit exceeded
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 16 Aug 2017 12:51:07 -0000


     [ https://issues.apache.org/jira/browse/HADOOP-14660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-14660:
------------------------------------
       Resolution: Fixed
    Fix Version/s: 3.0.0-beta1
                   2.9.0
           Status: Resolved  (was: Patch Available)

+1, committed

thanks for wrapping this up...closing the issue now

> wasb: improve throughput by 34% when account limit exceeded
> -----------------------------------------------------------
>
>                 Key: HADOOP-14660
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14660
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>            Reporter: Thomas Marquardt
>            Assignee: Thomas Marquardt
>             Fix For: 2.9.0, 3.0.0-beta1
>
>         Attachments: HADOOP-14660-001.patch, HADOOP-14660-002.patch, HADOOP-14660-003.patch, HADOOP-14660-004.patch, HADOOP-14660-005.patch, HADOOP-14660-006.patch, HADOOP-14660-007.patch, HADOOP-14660-008.patch, HADOOP-14660-010.patch, HADOOP-14660-branch-2-001.patch
>
>
> Big data workloads frequently exceed the Azure Storage max ingress and egress limits (https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits).  For example, the max ingress limit for a GRS account in the United States is currently 10 Gbps.  When the limit is exceeded, the Azure Storage service fails a percentage of incoming requests, and this causes the client to initiate the retry policy.  The retry policy delays requests by sleeping, but the sleep duration is independent of the client throughput and account limit.  This results in low throughput, due to the high number of failed requests and thrashing causes by the retry policy.
> To fix this, we introduce a client-side throttle which minimizes failed requests and maximizes throughput.  Tests have shown that this improves throughtput by ~34% when the storage account max ingress and/or egress limits are exceeded. 


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org