airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremiah Lowin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-62) XCom push not working reliably
Date Tue, 10 May 2016 01:35:12 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-62?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15277444#comment-15277444
] 

Jeremiah Lowin commented on AIRFLOW-62:
---------------------------------------

[~alex.papanic] I'm sorry to say this but so far I have not been able to replicate the issue.
I'm using a fresh install from airflow master, Postgres backend and LocalExecutor. I've been
trying to run CeleryExecutor but an unrelated error is interrupting me... could you try running
with LocalExecutor and see if that clears it up, in which case we can narrow this down to
a Celery issue?

> XCom push not working reliably
> ------------------------------
>
>                 Key: AIRFLOW-62
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-62
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: db, operators
>    Affects Versions: Airflow 1.7.0
>         Environment: Postgres backed Airflow running with Celery inside of the puckel
Docker setup.
>            Reporter: Alex Papanicolaou
>            Assignee: Jeremiah Lowin
>
> I have a DAG that polls for activity in various data streams from a database and then
uploads the activity statuses to a table.  Each of the polling tasks are python operators
that once they get the polling result, return a dict as an XCom push.  The dict contains two
entries which are strings, one which is a bool, and one which is a datetime object.  There
is a final task that pulls all the results and uploads the collective statuses to a table.
 I chose this pattern since I figured it might be better to do one collective write operation
on all the results.
> Before I moved ahead to the github master branch I was using 1.7.0 from PyPI and this
worked fine.  Now that I am on the github master branch, I find that the XCom pushing is unreliable.
 The returned values in the logs show up correctly but when doing the XCom pull, I get None
for some of the returned values.  Investigating the XCom result in the Webserver also shows
nothing there.  But if I rerun a task where the XCom failed, the push works and the XCom result
is as it should be.
> Nothing appears to have changed in the codebase so I am at a loss.  Perhaps it really
wasn't working before?  How would the backing postgres handle these simultaneous writes? 
I can't imagine that would be a problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message