hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "genericqa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15696) KMS performance regression due to too many open file descriptors after Jetty migration
Date Sat, 01 Sep 2018 00:48:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599439#comment-16599439
] 

genericqa commented on HADOOP-15696:
------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 24s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 23s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 35s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 23s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 25s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 40s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 27s{color}
| {color:green} branch has no errors when building and testing our client artifacts. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  1s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 54s{color} |
{color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 20s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 42s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m  9s{color} |
{color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 15m  9s{color} | {color:red}
root generated 1 new + 1454 unchanged - 0 fixed = 1455 total (was 1454) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  3m 57s{color}
| {color:orange} root: The patch generated 1 new + 91 unchanged - 0 fixed = 92 total (was
91) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 28s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  2s{color} | {color:green}
The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 13s{color}
| {color:green} patch has no errors when building and testing our client artifacts. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 15s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 54s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m 15s{color} | {color:green}
hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m  6s{color} | {color:green}
hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m  0s{color} | {color:green}
hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 41s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}124m 17s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15696 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12937865/HADOOP-15696.002.patch
|
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit
 shadedclient  findbugs  checkstyle  xml  |
| uname | Linux 148dbf1da6c2 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 50d2e3e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/15117/artifact/out/diff-compile-javac-root.txt
|
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/15117/artifact/out/diff-checkstyle-root.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/15117/testReport/ |
| Max. process+thread count | 1718 (vs. ulimit of 10000) |
| modules | C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs-httpfs
U: . |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15117/console |
| Powered by | Apache Yetus 0.9.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> KMS performance regression due to too many open file descriptors after Jetty migration
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-15696
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15696
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: kms
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Wei-Chiu Chuang
>            Assignee: Wei-Chiu Chuang
>            Priority: Blocker
>         Attachments: HADOOP-15696.001.patch, HADOOP-15696.002.patch, Screen Shot 2018-08-22
at 11.36.16 AM.png, Screen Shot 2018-08-22 at 4.26.51 PM.png, Screen Shot 2018-08-22 at 4.26.51
PM.png, Screen Shot 2018-08-22 at 4.27.02 PM.png, Screen Shot 2018-08-22 at 4.30.32 PM.png,
Screen Shot 2018-08-22 at 4.30.39 PM.png, Screen Shot 2018-08-24 at 7.08.16 PM.png
>
>
> We recently found KMS performance regressed in Hadoop 3.0, possibly linking to the migration
from Tomcat to Jetty in HADOOP-13597.
> Symptoms:
> # Hadoop 3.x KMS open file descriptors quickly rises to more than 10 thousand under stress,
sometimes even exceeds 32K, which is the system limit, causing failures for any access to
encryption zones. Our internal testing shows the openfd number was in the range of a few hundred
in Hadoop 2.x, and it increases by almost 100x in Hadoop 3.
> # Hadoop 3.x KMS as much as twice the heap size than in Hadoop 2.x. The same heap size
can go OOM in Hadoop 3.x. Jxray analysis suggests most of them are temporary byte arrays associated
with open SSL connections.
> # Due to the heap usage, Hadoop 3.x KMS has more frequent GC activities, and we observed
up to 20% performance reduction due to GC.
> A possible solution is to reduce the idle timeout setting in HttpServer2. It is currently
hard-coded 10 seconds. By setting it to 1 second, open fds dropped from 20 thousand down to
3 thousand in my experiment.
> File this jira to invite open discussion for a solution.
> Credit: [~misha@cloudera.com] for the proposed Jetty idle timeout remedy; [~xiaochen]
for digging into this problem.
> Screenshots:
> CDH5 (Hadoop 2) KMS CPU utilization, resident memory and file descriptor chart.
>  !Screen Shot 2018-08-22 at 4.30.39 PM.png! 
> CDH6 (Hadoop 3) KMS CPU utilization, resident memory and file descriptor chart.
>  !Screen Shot 2018-08-22 at 4.30.32 PM.png! 
> CDH5 (Hadoop 2) GC activities on the KMS process
>  !Screen Shot 2018-08-22 at 4.26.51 PM.png! 
> CDH6 (Hadoop 3) GC activities on the KMS process
>  !Screen Shot 2018-08-22 at 4.27.02 PM.png! 
> JXray report
>  !Screen Shot 2018-08-22 at 11.36.16 AM.png! 
> open fd drops from 20 k down to 3k after the proposed change.
>  !Screen Shot 2018-08-24 at 7.08.16 PM.png! 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message