Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
From: Bolke de Bruin <bdbruin@gmail.com>
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)
Date: Wed, 6 Dec 2017 17:17:58 +0100
Subject: Re: Xcom_push is broken - probably after the timezone awareness
Message-Id: <D24AF2FE-20DD-4BDD-BE39-549A6C9BE2B4@gmail.com>
References: <CAGeMxTcdnX9V60DeWSNN68qp5kXJCHi2JRPE-oJkk-8YeUEhCg@mail.gmail.com> <179836E3-F517-4E05-94C9-E65A2813D223@gmail.com> <CANRT7T3xHc0CBMBz0TWnBazdTQiY+eJuKV2nhuLjqyF2Juw8oQ@mail.gmail.com>
In-Reply-To: <CANRT7T3xHc0CBMBz0TWnBazdTQiY+eJuKV2nhuLjqyF2Juw8oQ@mail.gmail.com>
To: dev@airflow.incubator.apache.org
archived-at: Wed, 06 Dec 2017 16:18:07 -0000

1. Naive dates are supported for initalization of DAGs and will be, after th=
e patch of Sumit, for directly instantiated TaskInstances. XCom can store ar=
bitrary information so it is not affected.=20
2. Any arithmetic you do on dates will need to happen with a timezone associ=
ated with it. This rare, you normally do date + datetime. However if you wan=
t to compare dates you will need to make it timezone aware.
3. Timezone support will not be in 1.9.0=20
4. Logging.info will be supported in 1.9.0 final


So no reason to panic I would say ;-)

Bolke

Verstuurd vanaf mijn iPad

> Op 6 dec. 2017 om 16:35 heeft Boris Tyukin <boris@boristyukin.com> het vol=
gende geschreven:
>=20
> hm...can you elaborate why we cannot use  "from datetime import datetime".=

> it worked before just fine :) i was under an impression that any pickle
> serializable type can be using with xcom.
>=20
> Along with a recent change in logs (logging.info does not write to log
> files anymore), this makes upgrade to 1.9 quite difficult.
>=20
>> On Wed, Dec 6, 2017 at 9:19 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:=

>>=20
>> You should not use a naive datetime as it mentioned. So instead of of
>> using =E2=80=9Cfrom datetime import datetime=E2=80=9D use =E2=80=9Cfrom a=
irflow.utils.timezone
>> import datetime=E2=80=9D. This sets the timezone information that is requ=
ired to
>> the default configured in airflow.cfg.
>>=20
>> A patch that sets the default timezone for TaskInstance is also helpful i=
n
>> this case and will be welcomed. How to do it you can find in the DAG clas=
s.
>>=20
>> Bolke.
>>=20
>>> On 6 Dec 2017, at 15:13, Sumit Maheshwari <sumeet.manit@gmail.com>
>> wrote:
>>>=20
>>> Folks,
>>>=20
>>> I've some testes, which used to pass earlier, but recently they started
>> failing. When I looked closely it appeared that xcom_push is behaving
>> differently.
>>>=20
>>>=20
>>> So this is is the current code, which throws the error mentioned below
>> to it
>>> DEFAULT_DATE =3D datetime(2017, 1, 1)
>>> ti =3D TaskInstance(task=3Dtask, execution_date=3DDEFAULT_DATE)
>>> ti.xcom_push('abcd', 12345)
>>>=20
>>>  File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 31,
>> in process_bind_param
>>>    raise ValueError('naive datetime is disallowed')
>>> StatementError: (exceptions.ValueError) naive datetime is disallowed
>> [SQL: u'DELETE FROM xcom WHERE xcom."key" =3D ? AND xcom.execution_date =3D=
 ?
>> AND xcom.task_id =3D ? AND xcom.dag_id =3D ?'] [parameters: [{}]]
>>>=20
>>> When I changed the execution date param to a string, it fails with other=

>> error.
>>> ti =3D TaskInstance(task=3Dtask, execution_date=3DDEFAULT_DATE.isoformat=
())
>>> ti.xcom_push('abcd', 12345)
>>>=20
>>>  File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 29,
>> in process_bind_param
>>>    repr(value))
>>> StatementError: (exceptions.TypeError) expected datetime.datetime, not
>> '2017-01-01T00:00:00' [SQL: u'DELETE FROM xcom WHERE xcom."key" =3D ? AND=

>> xcom.execution_date =3D ? AND xcom.task_id =3D ? AND xcom.dag_id =3D ?']
>> [parameters: [{}]]
>>>=20
>>> Somehow I feel that it is linked with sqlalchemy_utc library, though I
>> may be wrong and its just missing some timezone related settings.
>>> Please let me know if anyone else has faced this issue as well.
>>>=20
>>> Thanks,
>>> Sumit
>>>=20
>>>=20
>>=20
>>=20