airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [airflow] kaxil commented on a change in pull request #12184: Create DAG-level cluster policy
Date Mon, 09 Nov 2020 20:13:47 GMT

kaxil commented on a change in pull request #12184:
URL: https://github.com/apache/airflow/pull/12184#discussion_r520091963



##########
File path: docs/concepts.rst
##########
@@ -1194,72 +1194,67 @@ state.
 
 Cluster Policy
 ==============
-Cluster policies provide an interface for taking action on every Airflow task
-either at DAG load time or just before task execution.
-
-Cluster Policies for Task Mutation
------------------------------------
-In case you want to apply cluster-wide mutations to the Airflow tasks,
-you can either mutate the task right after the DAG is loaded or
-mutate the task instance before task execution.
-
-Mutate tasks after DAG loaded
------------------------------
-
-To mutate the task right after the DAG is parsed, you can define
-a ``policy`` function in ``airflow_local_settings.py`` that mutates the
-task based on other task or DAG attributes (through ``task.dag``).
-It receives a single argument as a reference to the task object and you can alter
-its attributes.
 
-For example, this function could apply a specific queue property when
-using a specific operator, or enforce a task timeout policy, making sure
-that no tasks run for more than 48 hours. Here's an example of what this
-may look like inside your ``airflow_local_settings.py``:
+Cluster policies provide an interface for taking action on every Airflow task
+or DAG either at DAG load time or just before task execution. In this way users
+are able to do the following:
 
+- set default arguments on each DAG/task
+- checks that DAG/task meets required standards
+- perform custom logic of routing task to a queue
 
-.. code-block:: python
+Any many other options. To use cluster-wide policies users can define in their

Review comment:
       ```suggestion
   And many other options. To use cluster-wide policies users can define in their
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message