airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [airflow] notatallshaw edited a comment on issue #6370: AIRFLOW-5701: Don't clear xcom explicitly before execution
Date Sun, 27 Oct 2019 15:43:04 GMT
notatallshaw edited a comment on issue #6370: AIRFLOW-5701: Don't clear xcom explicitly before
execution
URL: https://github.com/apache/airflow/pull/6370#issuecomment-546706296
 
 
   @Fokko here's a proof of concept I am thinking of (requires Python 3.7+ to actually run)
that from what I understand would behave differently compared to now vs with this PR:
   
   ```python
   def execute(context):
       xcom_pull= context['ti'].xcom_pull
       xcom_push = context['ti'].xcom_pull
       execution_date = context['execution_date']
   
       old_xcom = xcom_pull(key='test', include_prior_dates=True)
       xcom_push(key='test', execution_date.isoformat())
       if old_xcom is None or datetime.fromisoformat(old_xcom) < execution_date:
           raise ValueError
   ```
   
   Currently if this was set to retry 3 times it would fail 3 times. But as I understand it
after this PR on the 2nd run `xcom_pull` will pull the value that was pushed from the previous
retry and the task will succeed. Am I missing something?
   
   Also it was stated that the xcom value is deleted when clearing a task instances, is it
also cleared when the user clicks "run" on that task instances? In my environment this is
the more common way of re-running a failed job.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message