helix-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HELIX-601) Allow work flow to schedule dependency jobs in parallel
Date Mon, 22 Jun 2015 04:27:00 GMT

    [ https://issues.apache.org/jira/browse/HELIX-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14595347#comment-14595347
] 

ASF GitHub Bot commented on HELIX-601:
--------------------------------------

Github user kanakb commented on a diff in the pull request:

    https://github.com/apache/helix/pull/30#discussion_r32904781
  
    --- Diff: helix-core/src/main/java/org/apache/helix/task/TaskRebalancer.java ---
    @@ -134,14 +134,22 @@ public ResourceAssignment computeBestPossiblePartitionState(ClusterDataCache
clu
           workflowCtx.setStartTime(System.currentTimeMillis());
         }
     
    -    // Check parent dependencies
    -    for (String parent : workflowCfg.getJobDag().getDirectParents(resourceName)) {
    -      if (workflowCtx.getJobState(parent) == null
    -          || !workflowCtx.getJobState(parent).equals(TaskState.COMPLETED)) {
    -        return emptyAssignment(resourceName, currStateOutput);
    +    // check ancestor job status
    +    int unStartCount = 0;
    --- End diff --
    
    Please rename to `notStartedCount` and `incompleteCount`


> Allow work flow to schedule dependency jobs in parallel
> -------------------------------------------------------
>
>                 Key: HELIX-601
>                 URL: https://issues.apache.org/jira/browse/HELIX-601
>             Project: Apache Helix
>          Issue Type: New Feature
>            Reporter: Congrui Ji
>
> Currently, Helix won't schedule dependency jobs in a same work flow. For example, if
Job2 depends on Job1, Job2 won't be scheduled until every partition of Job1 is completed.
> However, if some participant is very slow, then all dependency jobs is waiting for that
single participant.
> Helix should be able to schedule multiple jobs according to a parameter.
> A.C.
> 1. Introduce parallel count parameter in work flow and job queue.
> 2. Dependency jobs can be scheduled according to the parameter (Now the parameter is
always 1, so no parallel)
> 3. If Job2 depends on Job1, Job1 is scheduled before Job2.
> 4. No parallel jobs on the same instance. If a instance is running Job1, it won't run
Job2 until Job1 is finished. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message