airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sumit Maheshwari <sumeet.ma...@gmail.com>
Subject Re: Xcom_push is broken - probably after the timezone awareness
Date Wed, 06 Dec 2017 14:25:58 GMT
Thanks Bolke for the information.

BTW will existing dags which are using native datetime start failing after
this change, given no patch is done??

Just to be precise, dags which may be using methods like xcom push/pull?

On Dec 6, 2017 7:49 PM, "Bolke de Bruin" <bdbruin@gmail.com> wrote:

> You should not use a naive datetime as it mentioned. So instead of of
> using “from datetime import datetime” use “from airflow.utils.timezone
> import datetime”. This sets the timezone information that is required to
> the default configured in airflow.cfg.
>
> A patch that sets the default timezone for TaskInstance is also helpful in
> this case and will be welcomed. How to do it you can find in the DAG class.
>
> Bolke.
>
> On 6 Dec 2017, at 15:13, Sumit Maheshwari <sumeet.manit@gmail.com> wrote:
>
> Folks,
>
> I've some testes, which used to pass earlier, but recently they started
> failing. When I looked closely it appeared that xcom_push is behaving
> differently.
>
>
> So this is is the current code, which throws the error mentioned below to
> it
>
> DEFAULT_DATE = datetime(2017, 1, 1)
>
> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
> ti.xcom_push('abcd', 12345)
>
>
>   File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 31, in process_bind_param
>
>     raise ValueError('naive datetime is disallowed')
> StatementError: (exceptions.ValueError) naive datetime is disallowed [SQL: u'DELETE FROM
xcom WHERE xcom."key" = ? AND xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id
= ?'] [parameters: [{}]]
>
>
> When I changed the execution date param to a string, it fails with other
> error.
>
> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE.isoformat())
> ti.xcom_push('abcd', 12345)
>
>
>   File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 29, in process_bind_param
>     repr(value))
> StatementError: (exceptions.TypeError) expected datetime.datetime, not '2017-01-01T00:00:00'
[SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ? AND xcom.task_id
= ? AND xcom.dag_id = ?'] [parameters: [{}]]
>
>
> Somehow I feel that it is linked with *sqlalchemy_utc* library, though I may be wrong
and its just missing some timezone related settings.
>
> Please let me know if anyone else has faced this issue as well.
>
>
> Thanks,
>
> Sumit
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message