airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: [VOTE]: Using fractional seconds
Date Tue, 08 Nov 2016 18:40:12 GMT
I vote for option 1.

We may want to alter previous database migration script to have some
MySQL-specfic, `try` block to get it right on fresh installs.

We also may want a new database migration that is MysQL-specific and ALTERs
the columns properly. It seems to me thought that this might require high
level locks and take some time to execute on large tables (I'm thinking
`task_instance`). No one likes to see a database migration script hang for
minutes... An alternate approach might be for someone in the community to
share a script that does this and that people can review and decide whether
they want to run it, and perhaps when to run it, maybe after archiving some
of the large tables in their environment.

Max

On Tue, Nov 8, 2016 at 6:39 AM, Vishal Doshi <vishal@celect.com> wrote:

> We have an (atypical) use case where one DAG launches multiple runs of
> another DAG (but with different parameters). Without the precision, we have
> to add a second between each launch to avoid the database issues. Moving
> towards allowing fractional seconds would be great for us.
>
> Thanks,
> Vishal
>
> On 11/8/16, 04:29, "Bolke de Bruin" <bdbruin@gmail.com> wrote:
>
>     Dear All,
>
>     I’m trying to move over the testing infrastructure to the new
> infrastructure based on ubuntu 14.04 (we are on 12.04 now). 12.04 uses
> MySQL 5.5 and 14.04 allows the use of MySQL 5.6, which we say we are
> compatible with. MySQL does not store fractional seconds. Until version
> 5.6.4 (https://dev.mysql.com/doc/refman/5.6/en/fractional-seconds.html)
> it cuts off fractional seconds at comparison time, eg. comparing
> “2016-01-01 00:00:00.000001” against what is stored in MySQL “2016-01-01
> 00:00:00” would return a tuple in 5.6.4 but will fail beyond 5.6.4. The
> issue presents itself if you use the “@once” schedule interval.
>
>     Other databases (Postgres, SQLite, etc) store fractional seconds by
> default so do not exhibit this error. Since MySQL 5.6.4 it can also store
> fractional seconds, but for backwards compatibility it needs to be
> specified in the schema. Also note that MySQL behavior (not storing
> fractional seconds) goes against SQL standards as is noted by themselves (
> http://dev.mysql.com/doc/refman/5.7/en/fractional-seconds.html).
>
>     There are two solutions to this issue:
>
>     1. Update the schema for MySQL to include fractional seconds.
>     PRO:
>     - no coding changes
>     - makes mysql behave conform standards
>     - easier to maintain
>     - future proof
>
>     CON:
>     - needs to maintain schema
>     - requires an update to the schema of running mysql instances
>
>     2. Change the code to remove fractional settings (particularly .now()
> invocations)
>     PRO:
>     - No impact on running MySQL instances
>
>     CON:
>     - Impact on other databases that now loose precision, and might for a
> brief time show different behavior
>     - Code to maintain, cannot use .now() directly
>     - Be very careful when using date time and accessing the DB
>
>
>     There was some back and forth discussion on bitter about this, but we
> don’t seem to reach a conclusion. Hence I would like to call for a vote -
> at this election day :). Of course with arguments if needed. If there is a
> better way I’m of course open to that.
>
>
>     I vote for OPTION 1.
>
>     Bolke
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message