hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15703) ABFS - Implement client-side throttling
Date Wed, 05 Sep 2018 17:20:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604696#comment-16604696

Sean Mackrory commented on HADOOP-15703:

FYI, Yetus never ran on this. It introduced a findbugs warning (https://builds.apache.org/job/PreCommit-HADOOP-Build/15137/artifact/out/branch-findbugs-hadoop-tools_hadoop-azure-warnings.html)
and an ASF licensing issue (https://builds.apache.org/job/PreCommit-HADOOP-Build/15137/artifact/out/patch-asflicense-problems.txt).

> ABFS - Implement client-side throttling 
> ----------------------------------------
>                 Key: HADOOP-15703
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15703
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Sneha Varma
>            Assignee: Sneha Varma
>            Priority: Major
>         Attachments: HADOOP-15703-HADOOP-15407-001.patch, HADOOP-15703-HADOOP-15407-002.patch
> Big data workloads frequently exceed the AzureBlobFS max ingress and egress limits (https://docs.microsoft.com/en-us/azure/storage/common/storage-scalability-targets).
For example, the max ingress limit for a GRS account in the United States is currently 10
Gbps. When the limit is exceeded, the AzureBlobFS service fails a percentage of incoming requests,
and this causes the client to initiate the retry policy. The retry policy delays requests
by sleeping, but the sleep duration is independent of the client throughput and account limit.
This results in low throughput, due to the high number of failed requests and thrashing causes
by the retry policy.
> To fix this, we introduce a client-side throttle which minimizes failed requests and
maximizes throughput. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message