nemo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-nemo] johnyangk commented on a change in pull request #220: [NEMO-395] Address SonarCloud issues for the scheduler package
Date Fri, 14 Jun 2019 07:34:40 GMT
johnyangk commented on a change in pull request #220: [NEMO-395] Address SonarCloud issues
for the scheduler package
URL: https://github.com/apache/incubator-nemo/pull/220#discussion_r293690718
 
 

 ##########
 File path: runtime/master/src/main/java/org/apache/nemo/runtime/master/scheduler/BatchScheduler.java
 ##########
 @@ -447,6 +424,60 @@ private void doSchedule() {
     return tasks;
   }
 
+  ////////////////////////////////////////////////////////////////////// Task cloning methods.
+
+  /**
+   * @return true if the number of clones is modified.
+   *         false otherwise.
+   */
+  private boolean modifyStageNumClone(final Stage stage, final ClonedSchedulingProperty.CloneConf
cloneConf) {
+    final double fractionToWaitFor = cloneConf.getFractionToWaitFor();
+    final Object[] completedTaskTimes = planStateManager.getCompletedTaskTimeListMs(stage.getId()).toArray();
+
+    // Only after the fraction of the tasks are done...
+    // Delayed cloning (aggressive)
+    if (completedTaskTimes.length > 0
+      && completedTaskTimes.length >= Math.round(stage.getTaskIndices().size()
* fractionToWaitFor)) {
+      // Only if the running task is considered a 'straggler'....
+      Arrays.sort(completedTaskTimes);
+      final long medianTime = (long) completedTaskTimes[completedTaskTimes.length / 2];
+      final double medianTimeMultiplier = cloneConf.getMedianTimeMultiplier();
+      final Map<String, Long> executingTaskToTime = planStateManager.getExecutingTaskToRunningTimeMs(stage.getId());
+
+      return modifyStageNumCloneUsingMedianTime(
+        stage.getId(), completedTaskTimes.length, medianTime, medianTimeMultiplier, executingTaskToTime);
+    } else {
+      return false;
+    }
+  }
+
+  /**
+   * @return true if the number of clones is modified.
+   *         false otherwise.
+   */
+  private boolean modifyStageNumCloneUsingMedianTime(final String stageId,
 
 Review comment:
   We probably need this to avoid the "Cognitive Complexity" issue.
   
   Link to the SonarCloud issue.
   https://sonarcloud.io/project/issues?id=org.apache.nemo%3Anemo-project&issues=AWkltI_5gtFwgfOLu2KZ&open=AWkltI_5gtFwgfOLu2KZ

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message