airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: explicit_defaults_for_timestamp for mysql
Date Wed, 24 Oct 2018 05:09:46 GMT
We only need it at table creation time or alter table time during which an alembic script would
fail if MySQL restarts I assume?

I'm not sure if the PR in this way is required (but if it works and works well it's okay to
me too just like consistency across DBS and no surprises with MySQL )

Sent from my iPhone

> On 24 Oct 2018, at 05:18, Feng Lu <fenglu@google.com> wrote:
> 
> Sorry for the late reply. 
> GCP (CloudSQL) does support setting this parameter at session level but the VM used to
host the mysqld might be restarted at any time, so it can't be done reliably. 
> 
> Haotian (cc-ed) in my team has looked into the needed schema changes to make Airflow
1.10 timestamp support to work with mysql without setting the exlicit_defaults_for_timestamp
flag in mysql, details below: 
> 
> @@ -40,10 +40,6 @@
>      conn = op.get_bind()
>      if conn.dialect.name == 'mysql':
>          conn.execute("SET time_zone = '+00:00'")
> -        cur = conn.execute("SELECT @@explicit_defaults_for_timestamp")
> -        res = cur.fetchall()
> -        if res[0][0] == 0:
> -            raise Exception("Global variable explicit_defaults_for_timestamp needs to
be on (1) for mysql")
> 
>          op.alter_column(table_name='chart', column_name='last_modified', type_=mysql.TIMESTAMP(fsp=6))
> 
> @@ -69,20 +65,28 @@
>          op.alter_column(table_name='log', column_name='dttm', type_=mysql.TIMESTAMP(fsp=6))
>          op.alter_column(table_name='log', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6))
> 
> -        op.alter_column(table_name='sla_miss', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6),
nullable=False)
> +        op.alter_column(table_name='sla_miss', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6),
\
> +            nullable=False, server_default=sa.text('CURRENT_TIMESTAMP(6)'))
>          op.alter_column(table_name='sla_miss', column_name='timestamp', type_=mysql.TIMESTAMP(fsp=6))
> 
> -        op.alter_column(table_name='task_fail', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6))
> +        op.alter_column(table_name='task_fail', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6),
\
> +            nullable=False, server_default=sa.text('CURRENT_TIMESTAMP(6)'))
>          op.alter_column(table_name='task_fail', column_name='start_date', type_=mysql.TIMESTAMP(fsp=6))
>          op.alter_column(table_name='task_fail', column_name='end_date', type_=mysql.TIMESTAMP(fsp=6))
> 
> -        op.alter_column(table_name='task_instance', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6),
nullable=False)
> +        op.alter_column(table_name='task_instance', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6),
\
> +            nullable=False, server_default=sa.text('CURRENT_TIMESTAMP(6)'))
>          op.alter_column(table_name='task_instance', column_name='start_date', type_=mysql.TIMESTAMP(fsp=6))
>          op.alter_column(table_name='task_instance', column_name='end_date', type_=mysql.TIMESTAMP(fsp=6))
>          op.alter_column(table_name='task_instance', column_name='queued_dttm', type_=mysql.TIMESTAMP(fsp=6))
> 
> -        op.alter_column(table_name='xcom', column_name='timestamp', type_=mysql.TIMESTAMP(fsp=6))
> -        op.alter_column(table_name='xcom', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6))
> +        op.alter_column(table_name='xcom', column_name='timestamp', type_=mysql.TIMESTAMP(fsp=6),
\
> +            nullable=False, server_default=sa.text('CURRENT_TIMESTAMP(6)'))
> +        op.alter_column(table_name='xcom', column_name='execution_date', type_=mysql.TIMESTAMP(fsp=6),
\
> +            nullable=False, server_default=sa.text('CURRENT_TIMESTAMP(6)'))
> +        conn.execute("alter table task_instance alter column execution_date drop default")
> +        conn.execute("alter table sla_miss alter column execution_date drop default")
> +        conn.execute("alter table task_fail alter column execution_date drop default")
>      else:
>          # sqlite datetime is fine as is not converting
>          if conn.dialect.name == 'sqlite':
> --- migrations/versions/f23433877c24_fix_mysql_not_null_constraint.py
> +++ migrations/versions/f23433877c24_fix_mysql_not_null_constraint.py
> @@ -39,10 +39,15 @@
>      conn = op.get_bind()
>      if conn.dialect.name == 'mysql':
>          conn.execute("SET time_zone = '+00:00'")
> -        op.alter_column('task_fail', 'execution_date', existing_type=mysql.TIMESTAMP(fsp=6),
nullable=False)
> -        op.alter_column('xcom', 'execution_date', existing_type=mysql.TIMESTAMP(fsp=6),
nullable=False)
> -        op.alter_column('xcom', 'timestamp', existing_type=mysql.TIMESTAMP(fsp=6), nullable=False)
> -
> +        op.alter_column('task_fail', 'execution_date', existing_type=mysql.TIMESTAMP(fsp=6),
\
> +            nullable=False, server_default=sa.text('CURRENT_TIMESTAMP(6)'))
> +        op.alter_column('xcom', 'execution_date', existing_type=mysql.TIMESTAMP(fsp=6),
\
> +            nullable=False, server_default=sa.text('CURRENT_TIMESTAMP(6)'))
> +        op.alter_column('xcom', 'timestamp', existing_type=mysql.TIMESTAMP(fsp=6), \
> +            nullable=False, server_default=sa.text('CURRENT_TIMESTAMP(6)'))
> +        conn.execute("alter table task_fail alter column execution_date drop default")
> +        conn.execute("alter table xcom alter column execution_date drop default")
> +        conn.execute("alter table xcom alter column timestamp drop default")
> 
> He will make a PR for this soon. 
> 
> Feng 
> 
>> On Fri, Oct 19, 2018 at 10:26 AM Bolke de Bruin <bdbruin@gmail.com> wrote:
>> O and reading 
>> 
>> https://dev.mysql.com/doc/refman/5.6/en/server-system-variables.html#sysvar_explicit_defaults_for_timestamp
>> 
>> indicates that it can be set on the session level as well. So we could just change
the alembic scripts do try it. However
>> MariaDB does not support it in a session so we always need to check the variable.
We will also need to set it at *every* 
>> alembic script that deals with datetimes in the future. Nevertheless this might be
the easiest solution.
>> 
>> Does GCP’s MySQL also allow this setting in the session scope? 
>> 
>> B.
>> 
>>> On 19 Oct 2018, at 18:48, Deng Xiaodong <xd.deng.r@gmail.com> wrote:
>>> 
>>> I'm ok to test this.
>>> 
>>> @ash, may you kindly give some examples of what exact behaviour the testers
>>> should pay attention to? Since people like me may not know the full
>>> background of having introduced this restriction & check, or what issue it
>>> was trying to address.
>>> 
>>> @Feng Lu, may you please advise if you are still interested to prepare this
>>> PR?
>>> 
>>> Thanks!
>>> 
>>> 
>>> XD
>>> 
>>>> On Sat, Oct 20, 2018 at 12:38 AM Ash Berlin-Taylor <ash@apache.org>
wrote:
>>>> 
>>>> This sounds sensible and would mean we could also run on GCP's MySQL
>>>> offering too.
>>>> 
>>>> This would need someone to try out and check that timezones behave
>>>> sensibly with this change made.
>>>> 
>>>> Any volunteers?
>>>> 
>>>> -ash
>>>> 
>>>>> On 19 Oct 2018, at 17:32, Deng Xiaodong <xd.deng.r@gmail.com> wrote:
>>>>> 
>>>>> Wondering if there is any further thoughts about this proposal kindly
>>>> raised by Feng Lu earlier?
>>>>> 
>>>>> If we can skip this check & allow explicit_defaults_for_timestamp
to be
>>>> 0, it would be helpful, especially for enterprise users in whose
>>>> environments it’s really hard to ask for a database global variable change
>>>> (like myself…).
>>>>> 
>>>>> 
>>>>> XD
>>>>> 
>>>>>> On 2018/08/28 15:23:10, Feng Lu <f...@google.com.INVALID> wrote:
>>>>>> Bolke, a gentle ping..>
>>>>>> Thank you.>
>>>>>> 
>>>>>> On Thu, Aug 23, 2018, 23:01 Feng Lu <fe...@google.com> wrote:>
>>>>>> 
>>>>>>> Hi all,>
>>>>>>>> 
>>>>>>> After reading the MySQL documentation on the>
>>>>>>> exlicit_defaults_for_timestamp, it appears that we can skip the
check
>>>> on explicit_defaults_for_timestamp>
>>>>>>> = 1>
>>>>>>> <
>>>> https://github.com/apache/incubator-airflow/blob/master/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py#L43>
>>>> by>
>>>>>>> setting the column to accept NULL explicitly. For example:>
>>>>>>>> 
>>>>>>> op.alter_column(table_name='chart', column_name='last_modified',>
>>>>>>> type_=mysql.TIMESTAMP(fsp=6)) -->>
>>>>>>> op.alter_column(table_name='chart', column_name='last_modified',>
>>>>>>> type_=mysql.TIMESTAMP(fsp=6), nullable=True)>
>>>>>>>> 
>>>>>>> Here's why:>
>>>>>>> From MySQL doc (when explicit_defaults_for_timestamp is set to
True):>
>>>>>>> "TIMESTAMP columns not explicitly declared with the NOT NULL
attribute
>>>> are>
>>>>>>> automatically declared with the NULL attribute and permit NULL
>>>> values.>
>>>>>>> Assigning such a column a value of NULL sets it to NULL, not
the
>>>> current>
>>>>>>> timestamp.">
>>>>>>>> 
>>>>>>> Thanks and happy to shoot a PR if it makes sense.>
>>>>>>>> 
>>>>>>> Feng>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>> 
>>>> 
>> 

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message