hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yunjiong zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6285) Add option to set max limit on ResourceManager for ApplicationClientProtocol.getApplications
Date Mon, 13 Mar 2017 23:54:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923208#comment-15923208
] 

yunjiong zhao commented on YARN-6285:
-------------------------------------

Sorry for late response.
2.25 seconds in getApplications doesn't include ResourceRequest.
Most of the time was spend on getLogAggregationReportsForApp as stack trace shows.

I believe below code change should improve the performance (will test later)
{code}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
index ab283e7..926c757 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
@@ -296,6 +296,8 @@ public static ReservationRequestInterpreter convertFromProtoFormat(
    * Log Aggregation Status
    */
   private static final String LOG_AGGREGATION_STATUS_PREFIX = "LOG_";
+  private static final int LOG_AGGREGATION_STATUS_PREFIX_LEN =
+      LOG_AGGREGATION_STATUS_PREFIX.length();
   public static LogAggregationStatusProto convertToProtoFormat(
       LogAggregationStatus e) {
     return LogAggregationStatusProto.valueOf(LOG_AGGREGATION_STATUS_PREFIX
@@ -304,8 +306,8 @@ public static LogAggregationStatusProto convertToProtoFormat(
 
   public static LogAggregationStatus convertFromProtoFormat(
       LogAggregationStatusProto e) {
-    return LogAggregationStatus.valueOf(e.name().replace(
-      LOG_AGGREGATION_STATUS_PREFIX, ""));
+    return LogAggregationStatus.valueOf(e.name().substring(
+        LOG_AGGREGATION_STATUS_PREFIX_LEN));
   }
 
   /*
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
index 9f00b2e..1db66a5 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
@@ -1700,17 +1700,16 @@ public ResourceRequest getAMResourceRequest() {
       Map<NodeId, LogAggregationReport> outputs =
           new HashMap<NodeId, LogAggregationReport>();
       outputs.putAll(logAggregationStatus);
-      if (!isLogAggregationFinished()) {
+      if (!isLogAggregationFinished() && isAppInFinalState(this) &&
+          System.currentTimeMillis() > this.logAggregationStartTime
+          + this.logAggregationStatusTimeout) {
         for (Entry<NodeId, LogAggregationReport> output : outputs.entrySet()) {
           if (!output.getValue().getLogAggregationStatus()
             .equals(LogAggregationStatus.TIME_OUT)
               && !output.getValue().getLogAggregationStatus()
                 .equals(LogAggregationStatus.SUCCEEDED)
               && !output.getValue().getLogAggregationStatus()
-                .equals(LogAggregationStatus.FAILED)
-              && isAppInFinalState(this)
-              && System.currentTimeMillis() > this.logAggregationStartTime
-                  + this.logAggregationStatusTimeout) {
+                .equals(LogAggregationStatus.FAILED)) {
             output.getValue().setLogAggregationStatus(
               LogAggregationStatus.TIME_OUT);
           }
{code}
Should I open a new issue for those changes?

{quote}
1) Add parameter to indicate if we should include ResourceRequest/getLogAggregationReportsForApp
in the response, default is true to make it compatible. (Can be done if above experimental
shows it really helps).
{quote}
This will help if user use those parameters.


> Add option to set max limit on ResourceManager for ApplicationClientProtocol.getApplications
> --------------------------------------------------------------------------------------------
>
>                 Key: YARN-6285
>                 URL: https://issues.apache.org/jira/browse/YARN-6285
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: yunjiong zhao
>            Assignee: yunjiong zhao
>         Attachments: YARN-6285.001.patch, YARN-6285.002.patch, YARN-6285.003.patch
>
>
> When users called ApplicationClientProtocol.getApplications, it will return lots of data,
and generate lots of garbage on ResourceManager which caused long time GC.
> For example, on one of our RM, when called rest API " http://<rm http address:port>/ws/v1/cluster/apps"
it can return 150MB data which have 944 applications.
> getApplications have limit parameter, but some user might not set it, and then the limit
will be Long.MAX_VALUE.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message