hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-6339) Improve performance for createAndGetApplicationReport
Date Tue, 21 Mar 2017 20:18:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935235#comment-15935235
] 

Wangda Tan edited comment on YARN-6339 at 3/21/17 8:18 PM:
-----------------------------------------------------------

[~zhaoyunjiong], thanks for explanation, generally make sense to me.

Few more suggestions:

1) Can we make use of {{isLogAggregationFinished}} in {{createAndGetApplicationReport}} to
avoid following checks:
{code}
      if (LogAggregationStatus.FAILED == logAggregationStatusApp ||
          LogAggregationStatus.SUCCEEDED == logAggregationStatusApp ||
          LogAggregationStatus.TIME_OUT == logAggregationStatusApp) {
        try {
          this.writeLock.lock();
          this.logAggregationStatusForAppReport = logAggregationStatusApp;
        } finally {
          this.writeLock.unlock();
        }
      }
{code}

2) Can we make logAggregationStatusForAppReport to be volatile to avoid the additional writelock
of {{createAndGetApplicationReport}}?



was (Author: leftnoteasy):
[~zhaoyunjiong], thanks for explanation, generally make sense to me.

Few more suggestions:

1) Can we make use of {{isLogAggregationFinished}} in {{createAndGetApplicationReport}} to
avoid following checks:
{code}
      if (LogAggregationStatus.FAILED == logAggregationStatusApp ||
          LogAggregationStatus.SUCCEEDED == logAggregationStatusApp ||
          LogAggregationStatus.TIME_OUT == logAggregationStatusApp) {
        try {
          this.writeLock.lock();
          this.logAggregationStatusForAppReport = logAggregationStatusApp;
        } finally {
          this.writeLock.unlock();
        }
      }
{code}

2) Can we make logAggregationStatusForAppReport to be volatile to avoid the additional writelock?


> Improve performance for createAndGetApplicationReport
> -----------------------------------------------------
>
>                 Key: YARN-6339
>                 URL: https://issues.apache.org/jira/browse/YARN-6339
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: yunjiong zhao
>            Assignee: yunjiong zhao
>         Attachments: YARN-6339.001.patch, YARN-6339.002.patch
>
>
> There are two performance issue when calling createAndGetApplicationReport:
> One is inside ProtoUtils.convertFromProtoFormat, replace is too slow for clusters which
have more than 3000 nodes. Use substring is much better: https://issues.apache.org/jira/browse/YARN-6285?focusedCommentId=15923241&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15923241
> Another one is inside getLogAggregationReportsForApp, if some application's LogAggregationStatus
is TIME_OUT, every time it was called it will create an HashMap which will produce lots of
garbage.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message