airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Xcom_push is broken - probably after the timezone awareness
Date Wed, 06 Dec 2017 14:28:40 GMT
You are directly instantiating a TaskInstance which does not verify against naive dates being
passed, not a DAG. DAGs with naive dates will work fine.

So for TaskInstances, instantiated outside Airflow itself (ie your python file) it will fail
on master given no patch. So, I guess a patch is very welcome if you are relying on this behaviour.

Cheers
Bolke

> On 6 Dec 2017, at 15:25, Sumit Maheshwari <sumeet.manit@gmail.com> wrote:
> 
> Thanks Bolke for the information.
> 
> BTW will existing dags which are using native datetime start failing after this change,
given no patch is done??
> 
> Just to be precise, dags which may be using methods like xcom push/pull?
> 
> On Dec 6, 2017 7:49 PM, "Bolke de Bruin" <bdbruin@gmail.com <mailto:bdbruin@gmail.com>>
wrote:
> You should not use a naive datetime as it mentioned. So instead of of using “from datetime
import datetime” use “from airflow.utils.timezone import datetime”. This sets the timezone
information that is required to the default configured in airflow.cfg.
> 
> A patch that sets the default timezone for TaskInstance is also helpful in this case
and will be welcomed. How to do it you can find in the DAG class.
> 
> Bolke.
> 
>> On 6 Dec 2017, at 15:13, Sumit Maheshwari <sumeet.manit@gmail.com <mailto:sumeet.manit@gmail.com>>
wrote:
>> 
>> Folks, 
>> 
>> I've some testes, which used to pass earlier, but recently they started failing.
When I looked closely it appeared that xcom_push is behaving differently. 
>> 
>> 
>> So this is is the current code, which throws the error mentioned below to it
>> DEFAULT_DATE = datetime(2017, 1, 1)
>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
>> ti.xcom_push('abcd', 12345)
>> 
>>   File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 31, in process_bind_param
>>     raise ValueError('naive datetime is disallowed')
>> StatementError: (exceptions.ValueError) naive datetime is disallowed [SQL: u'DELETE
FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id
= ?'] [parameters: [{}]]
>> 
>> When I changed the execution date param to a string, it fails with other error. 
>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE.isoformat())
>> ti.xcom_push('abcd', 12345)
>> 
>>   File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 29, in process_bind_param
>>     repr(value))
>> StatementError: (exceptions.TypeError) expected datetime.datetime, not '2017-01-01T00:00:00'
[SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ? AND xcom.task_id
= ? AND xcom.dag_id = ?'] [parameters: [{}]]
>> 
>> Somehow I feel that it is linked with sqlalchemy_utc library, though I may be wrong
and its just missing some timezone related settings. 
>> Please let me know if anyone else has faced this issue as well. 
>> 
>> Thanks,
>> Sumit
>> 
>> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message