airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Papanicolaou (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-62) XCom push not working reliably
Date Tue, 10 May 2016 02:42:12 GMT


Alex Papanicolaou commented on AIRFLOW-62:

But I have no idea what the problem could be... Airflow is correctly using the scoped_session
that allows Celery and SQLAlchemy to play nice.  What could possibly be the problem? The whole
package would be unfunctional if the general setup of Celery and SQLAlchemy wasn't proper.

> XCom push not working reliably
> ------------------------------
>                 Key: AIRFLOW-62
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: db, operators
>    Affects Versions: Airflow 1.7.0
>         Environment: Postgres backed Airflow running with Celery inside of the puckel
Docker setup.
>            Reporter: Alex Papanicolaou
>            Assignee: Jeremiah Lowin
> I have a DAG that polls for activity in various data streams from a database and then
uploads the activity statuses to a table.  Each of the polling tasks are python operators
that once they get the polling result, return a dict as an XCom push.  The dict contains two
entries which are strings, one which is a bool, and one which is a datetime object.  There
is a final task that pulls all the results and uploads the collective statuses to a table.
 I chose this pattern since I figured it might be better to do one collective write operation
on all the results.
> Before I moved ahead to the github master branch I was using 1.7.0 from PyPI and this
worked fine.  Now that I am on the github master branch, I find that the XCom pushing is unreliable.
 The returned values in the logs show up correctly but when doing the XCom pull, I get None
for some of the returned values.  Investigating the XCom result in the Webserver also shows
nothing there.  But if I rerun a task where the XCom failed, the push works and the XCom result
is as it should be.
> Nothing appears to have changed in the codebase so I am at a loss.  Perhaps it really
wasn't working before?  How would the backing postgres handle these simultaneous writes? 
I can't imagine that would be a problem.

This message was sent by Atlassian JIRA

View raw message