crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao Shi (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-307) Limit the number of concurrently running jobs
Date Thu, 28 Nov 2013 03:08:35 GMT


Chao Shi updated CRUNCH-307:

    Attachment: crunch-307-v2.patch

My previous patch is wrong. This is found on the cluster test. It drops some ready tasks if
the "if..break" path is executed. I also added the test for such case.

> Limit the number of concurrently running jobs
> ---------------------------------------------
>                 Key: CRUNCH-307
>                 URL:
>             Project: Crunch
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Chao Shi
>            Assignee: Chao Shi
>         Attachments: crunch-307-v2.patch, crunch-307.patch
> In current behavior, crunch keep submitting new jobs as long as their dependencies have
finished. This is not good on a scenario where a pipeline consists a large number (e.g. 10+)
of independent DAGs. A improvement is to define a limited window of concurrent running jobs.
Whenever such limit is reached, we stop submitting new jobs and wait until any running job

This message was sent by Atlassian JIRA

View raw message