airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <>
Subject [GitHub] [airflow] dstandish commented on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator
Date Sun, 24 Nov 2019 21:45:54 GMT
dstandish commented on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator
   So there were concerns with @Fokko's xcom change re idempotency.
   I think it makes sense to create second table, very similar to xcom, but designed specifically
to support stateful tasks.  The table could perhaps be called TaskState.  
   This task state should not be pegged to a specific execution date, because execution date
only really makes sense for non-stateful tasks.  And execution date can be out of sequence
with actual run time.
   I think it might make sense to make it so we don't do updates: when state changes, we insert
a new record with the current state.  Primary key would be dag id / task id / timestamp. 
To get current state, we get the last record for the dag / task.  It's possible we could allow
state to be namespaced under task id with a column `key` like is done with XCom but I don't
think it's necessary. 
   I previously shared the concern, why create another table that is almost identical to xcom.
 But the reality is XCom is problematic for stateful tasks in a number of ways.  Obviously
there is the clearing / idempotency issue. But additionally if you use trigger dag, with XCom
your next scheduled run won't get current state because it sorts by execution_date.

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

With regards,
Apache Git Services

View raw message