airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pierrick Boitel (Jira)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-5295) DAG doesn't take the last value of XCOM
Date Fri, 06 Sep 2019 09:22:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-5295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16924077#comment-16924077
] 

Pierrick Boitel commented on AIRFLOW-5295:
------------------------------------------

running_cluster is took from an Airflow's variable. We use this for test to save time when
we alreday have a running cluster. But in general (on production), this variable is not set
so we takes the variable from the XCOM.
Thanks for the information to see the xcom took in tasks, didn't know this. Unfortunately
I don't have the history for the last time when the problems happens (it is only when we have
problem in our DAG). I will wait for the problem to happen again, and give you this information.
And if we don't have the problem in two months, I propose you to close the ticket if it's
ok for you (have put a reminder on my calendar to comment again on this ticket at this time).

> DAG doesn't take the last value of XCOM
> ---------------------------------------
>
>                 Key: AIRFLOW-5295
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5295
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: DAG
>    Affects Versions: 1.8.0, 1.10.3
>            Reporter: Pierrick Boitel
>            Priority: Minor
>
> I have sometimes failure on my DAG and need to re-run some tasks. Some of these emits
a new value for a XCOM, but when following tasks runs, they generally takes the old value
for this XCOM (although the value is good in Airflow's database). In order to have the good
value, I have to clear my tasks several time (more than 10 time sometimes) which is really
painfull.
> I suppose that the DAG on the scheduler keeps the value on cache but not sure on this.
I didn't enable pickking for the JSON too.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Mime
View raw message