airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michelle Huang (JIRA)" <>
Subject [jira] [Created] (AIRFLOW-1832) xcom_push is not reliable
Date Mon, 20 Nov 2017 15:31:00 GMT
Michelle Huang created AIRFLOW-1832:

             Summary: xcom_push is not reliable
                 Key: AIRFLOW-1832
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: Michelle Huang

we have a few ETL jobs that are hitting a third party API, extracting data onto the Airflow
server and then transforming and inserting that data into Redshift. we're using xcom_push
in the extract function to store an identifier appended to the filename to be referenced in
the transform function so we can grab the right file to transform. I'm noticing that xcom_push
is not successfully pushing the value and xcom_pull is returning "None". this is happening
across many of our scheduled jobs regularly and only just started happening on 11/16. manual
triggers of the jobs run successfully to completion without issue. this only affects our scheduled
runs and doesn't happen to all of them.

This message was sent by Atlassian JIRA

View raw message