crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao Shi (JIRA)" <>
Subject [jira] [Created] (CRUNCH-307) Limit the number of concurrently running jobs
Date Wed, 27 Nov 2013 03:41:35 GMT
Chao Shi created CRUNCH-307:

             Summary: Limit the number of concurrently running jobs
                 Key: CRUNCH-307
             Project: Crunch
          Issue Type: New Feature
          Components: Core
            Reporter: Chao Shi
            Assignee: Chao Shi
         Attachments: crunch-307.patch

In current behavior, crunch keep submitting new jobs as long as their dependencies have finished.
This is not good on a scenario where a pipeline consists a large number (e.g. 10+) of independent
DAGs. A improvement is to define a limited window of concurrent running jobs. Whenever such
limit is reached, we stop submitting new jobs and wait until any running job finishes.

This message was sent by Atlassian JIRA

View raw message