airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robin Miller (JIRA)" <>
Subject [jira] [Closed] (AIRFLOW-657) MsSqlOperator Autocommit Option
Date Mon, 18 Sep 2017 09:59:00 GMT


Robin Miller closed AIRFLOW-657.
    Resolution: Fixed

> MsSqlOperator Autocommit Option
> -------------------------------
>                 Key: AIRFLOW-657
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: operators
>            Reporter: Robin Miller
>            Assignee: Robin Miller
>            Priority: Trivial
>   Original Estimate: 1h
>  Remaining Estimate: 1h
> The MsSqlOperator is not exposing the option to enable autocommit, even though the MsSqlHook
has supports_autocommit to true.
> This means that tasks running on MsSql are always wrapped in an additional transaction,
which means that parallel tasks using shared resources can cause SQL Deadlocks.
> We see no reason not to expose this option to the user via the operator and have produced
and tested the change, keeping the existing behaviour as the default.
> In our use case we have a number of identical parallel tasks to work through a list of
identical build tasks with an appropriate level of parallelism. This is prevented from working
as intended since the first of these tasks to start, updates this task list and then holds
a transaction lock on this table, meaning the other threads cannot then access the table.
(It would not be convenient to have a task for each task in the list defined in the dag as
there are upwards of 7000 tasks and it is convenient for us to be able to manually add or
remove tasks from this list during execution)

This message was sent by Atlassian JIRA

View raw message