airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-341) Add Resources To Tasks
Date Tue, 19 Jul 2016 23:42:20 GMT


ASF subversion and git services commented on AIRFLOW-341:

Commit 348f25f08af2c02627cec04453564edd2fb69fa3 in incubator-airflow's branch refs/heads/master
from [~aoen]
[;h=348f25f ]

[AIRFLOW-341][operators] Add resource requirement attributes to operators

This PR adds optional resource requirements for tasks for use with
resource managers such as Yarn and Mesos.

- I chose to force users to encapsulate resources in a resources object
e.g. Resources(cpu=1) instead of just cpu=1 in their dag attributes.
This creates the pain of having to import Resources for almost every
DAG. I think this is kind of important for scoping/namespacing which we
should start doing.
- Once resources are used by executors we need to add documentation for
these new resources (and examples)

Testing Done:
- New/existing unit tests

plypaul artwr mistercrunch jlowin bolkedebruin criccomini

Closes #1669 from aoen/ddavydov/ddavydov/augment_tasks_with_resources

> Add Resources To Tasks
> ----------------------
>                 Key: AIRFLOW-341
>                 URL:
>             Project: Apache Airflow
>          Issue Type: New Feature
>            Reporter: Dan Davydov
>            Assignee: Dan Davydov
> Tasks should have optional resource requirements for use with resource managers such
as yarn and mesos.

This message was sent by Atlassian JIRA

View raw message