airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bence Nagy (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-30) Make preoperators part of the same transaction as the actual operation
Date Mon, 02 May 2016 09:21:12 GMT
Bence Nagy created AIRFLOW-30:
---------------------------------

             Summary: Make preoperators part of the same transaction as the actual operation
                 Key: AIRFLOW-30
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-30
             Project: Apache Airflow
          Issue Type: Improvement
            Reporter: Bence Nagy


All my use cases would work better if each operator would execute everything in one transaction.
Two examples:

- I want to {{GenericTransfer}} a set of rows from one DB to another, and I have to create
the table first in the destination DB. I feel like it'd be a lot more clean if I didn't have
empty tables lying around if the insertion fails for some reason later on.

- I want to {{GenericTransfer}} all rows from an entire table periodically to sync it from
one DB to another. To do this correctly I want to clear the destination table first to make
sure I end up with no duplicate rows, so I'd have a {{DELETE * FROM dst_table}} preoperator.
If the insertions fail afterwards, I'd end up with no data (it would be better in most cases
to fall back to the old data), and even if everything is working correctly, I'll have an empty
table while the insertions as still executing.

To fix this, the relevant {{DbApiHook}} methods could support a new kwarg to set whether it
should commit at the end.

Thoughts?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message