airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Riccomini (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (AIRFLOW-1619) Add sleep time parameter for GCP dataflow hook
Date Mon, 18 Sep 2017 18:25:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-1619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Riccomini resolved AIRFLOW-1619.
--------------------------------------
       Resolution: Fixed
    Fix Version/s: 1.10.0

> Add sleep time parameter for GCP dataflow hook
> ----------------------------------------------
>
>                 Key: AIRFLOW-1619
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1619
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib, Dataflow, gcp, hooks
>            Reporter: Thomas Buida
>            Assignee: Thomas Buida
>            Priority: Minor
>             Fix For: 1.10.0
>
>
> In the GCP dataflow hook, the sleep time while waiting for the dataflow job to be done
is hard coded to 10 seconds. For long running jobs (hours or even several days), this rate
is overkill when a poll time of several minutes or even an hour may be sufficient.
> Adding a poll_sleep parameter to the GCP dataflow operator that is used in the hook would
allow users to set a more sensible polling interval for long running jobs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message