airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Buida (JIRA)" <>
Subject [jira] [Created] (AIRFLOW-1619) Add sleep time parameter for GCP dataflow hook
Date Mon, 18 Sep 2017 13:56:00 GMT
Thomas Buida created AIRFLOW-1619:

             Summary: Add sleep time parameter for GCP dataflow hook
                 Key: AIRFLOW-1619
             Project: Apache Airflow
          Issue Type: Improvement
          Components: contrib, Dataflow, gcp, hooks
            Reporter: Thomas Buida
            Assignee: Thomas Buida
            Priority: Minor

In the GCP dataflow hook, the sleep time while waiting for the dataflow job to be done is
hard coded to 10 seconds. For long running jobs (hours or even several days), this rate is
overkill when a poll time of several minutes or even an hour may be sufficient.

Adding a poll_sleep parameter to the GCP dataflow operator that is used in the hook would
allow users to set a more sensible polling interval for long running jobs.

This message was sent by Atlassian JIRA

View raw message