hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14660) wasb: improve throughput by 34% when account limit exceeded
Date Mon, 14 Aug 2017 21:51:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126472#comment-16126472
] 

Hadoop QA commented on HADOOP-14660:
------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 19s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  1s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 4 new or modified test files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 45s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 40s{color}
| {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 26s{color} |
{color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 46s{color} |
{color:green} branch-2 passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 28s{color}
| {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 23s{color} |
{color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 18s{color} |
{color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  4s{color} |
{color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 10s{color} |
{color:green} branch-2 passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 17s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 15s{color} | {color:red}
hadoop-azure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  5m 39s{color} | {color:red}
root in the patch failed with JDK v1.8.0_144. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  5m 39s{color} | {color:red}
root in the patch failed with JDK v1.8.0_144. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  6m 35s{color} | {color:red}
root in the patch failed with JDK v1.7.0_131. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  6m 35s{color} | {color:red}
root in the patch failed with JDK v1.7.0_131. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  1m 27s{color}
| {color:orange} root: The patch generated 2 new + 3 unchanged - 22 fixed = 5 total (was 25)
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 20s{color} | {color:red}
hadoop-azure in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 20s{color} | {color:red}
hadoop-azure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 17s{color} | {color:red}
hadoop-tools_hadoop-azure-jdk1.8.0_144 with JDK v1.8.0_144 generated 4 new + 0 unchanged -
0 fixed = 4 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 19s{color} | {color:red}
hadoop-tools_hadoop-azure-jdk1.7.0_131 with JDK v1.7.0_131 generated 4 new + 1 unchanged -
0 fixed = 5 total (was 1) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 35s{color} | {color:green}
hadoop-common in the patch passed with JDK v1.7.0_131. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 20s{color} | {color:red}
hadoop-azure in the patch failed with JDK v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 27s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m  8s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:5e40efe |
| JIRA Issue | HADOOP-14660 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881813/HADOOP-14660-branch-2.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux 915e5a2eb21f 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / e5ba40c |
| Default Java | 1.7.0_131 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
|
| findbugs | v3.0.0 |
| mvninstall | https://builds.apache.org/job/PreCommit-HADOOP-Build/13031/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-azure.txt
|
| compile | https://builds.apache.org/job/PreCommit-HADOOP-Build/13031/artifact/patchprocess/patch-compile-root-jdk1.8.0_144.txt
|
| javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/13031/artifact/patchprocess/patch-compile-root-jdk1.8.0_144.txt
|
| compile | https://builds.apache.org/job/PreCommit-HADOOP-Build/13031/artifact/patchprocess/patch-compile-root-jdk1.7.0_131.txt
|
| javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/13031/artifact/patchprocess/patch-compile-root-jdk1.7.0_131.txt
|
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13031/artifact/patchprocess/diff-checkstyle-root.txt
|
| mvnsite | https://builds.apache.org/job/PreCommit-HADOOP-Build/13031/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-azure.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-HADOOP-Build/13031/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-azure.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-HADOOP-Build/13031/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-azure-jdk1.8.0_144.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-HADOOP-Build/13031/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-azure-jdk1.7.0_131.txt
|
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/13031/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure-jdk1.7.0_131.txt
|
| JDK v1.7.0_131  Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/13031/testReport/
|
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: . |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/13031/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> wasb: improve throughput by 34% when account limit exceeded
> -----------------------------------------------------------
>
>                 Key: HADOOP-14660
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14660
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>            Reporter: Thomas Marquardt
>            Assignee: Thomas Marquardt
>         Attachments: HADOOP-14660-001.patch, HADOOP-14660-002.patch, HADOOP-14660-003.patch,
HADOOP-14660-004.patch, HADOOP-14660-005.patch, HADOOP-14660-006.patch, HADOOP-14660-007.patch,
HADOOP-14660-008.patch, HADOOP-14660-010.patch, HADOOP-14660-branch-2.patch
>
>
> Big data workloads frequently exceed the Azure Storage max ingress and egress limits
(https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits).  For example,
the max ingress limit for a GRS account in the United States is currently 10 Gbps.  When the
limit is exceeded, the Azure Storage service fails a percentage of incoming requests, and
this causes the client to initiate the retry policy.  The retry policy delays requests by
sleeping, but the sleep duration is independent of the client throughput and account limit.
 This results in low throughput, due to the high number of failed requests and thrashing causes
by the retry policy.
> To fix this, we introduce a client-side throttle which minimizes failed requests and
maximizes throughput.  Tests have shown that this improves throughtput by ~34% when the storage
account max ingress and/or egress limits are exceeded. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message