airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Tyukin <bo...@boristyukin.com>
Subject Re: Xcom_push is broken - probably after the timezone awareness
Date Wed, 06 Dec 2017 18:20:32 GMT
:) you rock as always Bolke!!

On Wed, Dec 6, 2017 at 11:17 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:

> 1. Naive dates are supported for initalization of DAGs and will be, after
> the patch of Sumit, for directly instantiated TaskInstances. XCom can store
> arbitrary information so it is not affected.
> 2. Any arithmetic you do on dates will need to happen with a timezone
> associated with it. This rare, you normally do date + datetime. However if
> you want to compare dates you will need to make it timezone aware.
> 3. Timezone support will not be in 1.9.0
> 4. Logging.info will be supported in 1.9.0 final
>
>
> So no reason to panic I would say ;-)
>
> Bolke
>
> Verstuurd vanaf mijn iPad
>
> > Op 6 dec. 2017 om 16:35 heeft Boris Tyukin <boris@boristyukin.com> het
> volgende geschreven:
> >
> > hm...can you elaborate why we cannot use  "from datetime import
> datetime".
> > it worked before just fine :) i was under an impression that any pickle
> > serializable type can be using with xcom.
> >
> > Along with a recent change in logs (logging.info does not write to log
> > files anymore), this makes upgrade to 1.9 quite difficult.
> >
> >> On Wed, Dec 6, 2017 at 9:19 AM, Bolke de Bruin <bdbruin@gmail.com>
> wrote:
> >>
> >> You should not use a naive datetime as it mentioned. So instead of of
> >> using “from datetime import datetime” use “from airflow.utils.timezone
> >> import datetime”. This sets the timezone information that is required to
> >> the default configured in airflow.cfg.
> >>
> >> A patch that sets the default timezone for TaskInstance is also helpful
> in
> >> this case and will be welcomed. How to do it you can find in the DAG
> class.
> >>
> >> Bolke.
> >>
> >>> On 6 Dec 2017, at 15:13, Sumit Maheshwari <sumeet.manit@gmail.com>
> >> wrote:
> >>>
> >>> Folks,
> >>>
> >>> I've some testes, which used to pass earlier, but recently they started
> >> failing. When I looked closely it appeared that xcom_push is behaving
> >> differently.
> >>>
> >>>
> >>> So this is is the current code, which throws the error mentioned below
> >> to it
> >>> DEFAULT_DATE = datetime(2017, 1, 1)
> >>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
> >>> ti.xcom_push('abcd', 12345)
> >>>
> >>>  File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 31,
> >> in process_bind_param
> >>>    raise ValueError('naive datetime is disallowed')
> >>> StatementError: (exceptions.ValueError) naive datetime is disallowed
> >> [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date =
> ?
> >> AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]]
> >>>
> >>> When I changed the execution date param to a string, it fails with
> other
> >> error.
> >>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE.isoformat())
> >>> ti.xcom_push('abcd', 12345)
> >>>
> >>>  File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 29,
> >> in process_bind_param
> >>>    repr(value))
> >>> StatementError: (exceptions.TypeError) expected datetime.datetime, not
> >> '2017-01-01T00:00:00' [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND
> >> xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?']
> >> [parameters: [{}]]
> >>>
> >>> Somehow I feel that it is linked with sqlalchemy_utc library, though I
> >> may be wrong and its just missing some timezone related settings.
> >>> Please let me know if anyone else has faced this issue as well.
> >>>
> >>> Thanks,
> >>> Sumit
> >>>
> >>>
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message