airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arnab (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1002) Add ability to remove DAG and all dependencies
Date Sun, 15 Oct 2017 02:38:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204992#comment-16204992
] 

Arnab commented on AIRFLOW-1002:
--------------------------------

I was about to file a new JIRA for deletion of dags but noticed this one exists. So updating
this jira with my comments.

- Extended code for delete endpoint to handle non-existing DAG case (return 404)
- Support for subdags 
  The way I read the requirement is if a dag has subdags (dag id for subdag is <dag_id>.<task_id>),
then the subdag and its associated data should get deleted a
s well when the user deletes the parent dag.
  Deleting the subdag by itself should also be supported
- Also, if the scheduler is actively generating tasks for a dag, thought it may
be a good idea to pause the dag (on which delete is called) first to prevent any
 additional tasks being added and then delete the same.
- Considered additional tables to remove the dag record from (such as Xcom)
- Use 204(no content) as return code for delete
- Added 3 testcases(test_delete_invalid_dag, test_delete_dag_after_schedule_dag and test_delete_dag_after_schedule_dag)
   
  The first two tests are passing. The third one seems to be ok but is failing i
n the PR. Any help appreciated. 

  Submitted PR : https://github.com/apache/incubator-airflow/pull/2199

> Add ability to remove DAG and all dependencies 
> -----------------------------------------------
>
>                 Key: AIRFLOW-1002
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1002
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: core, DAG
>    Affects Versions: Airflow 2.0, Airflow 1.8
>            Reporter: Ruslan Dautkhanov
>            Assignee: Kengo Seki
>
> I don't see an obvious way to remove a DAG with dependencies
> Looks like folks have custom scripts to clean up backend database directly.
> http://stackoverflow.com/questions/40651783/airflow-how-to-delete-a-dag
> https://groups.google.com/forum/#!topic/airbnb_airflow/GVsNsUxPRC0
> It would be nice to have a feature to remove a DAG directly in UI / through API.
> {code}
> queries = {'delete from xcom where dag_id = "' + dag_input + '"',
>         'delete from task_instance where dag_id = "' + dag_input + '"',
>         'delete from sla_miss where dag_id = "' + dag_input + '"',
>         'delete from log where dag_id = "' + dag_input + '"',
>         'delete from job where dag_id = "' + dag_input + '"',
>         'delete from dag_run where dag_id = "' + dag_input + '"',
>         'delete from dag where dag_id = "' + dag_input + '"' }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message