airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Költringer (JIRA) <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-2329) airflow initdb fails: Specified key was too long; max key length is 767 bytes
Date Tue, 17 Apr 2018 09:21:00 GMT
Andreas Költringer created AIRFLOW-2329:
-------------------------------------------

             Summary: airflow initdb fails: Specified key was too long; max key length is
767 bytes 
                 Key: AIRFLOW-2329
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2329
             Project: Apache Airflow
          Issue Type: Bug
          Components: db
    Affects Versions: 1.9.0
            Reporter: Andreas Költringer


Turns out that the default charset in MariaDB is {{utf8mb4}}, and the default max. keylength
is 767 bytes ([MariaDB < 10.2.2|https://mariadb.com/kb/en/library/xtradbinnodb-server-system-variables/#innodb_large_prefix]
// [MySQL < 5.7.7|https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_large_prefix]
). The field {{dag_id}} is defined as {{VARCHAR(250)}}, {{250 x 4 = 1000 > 767}}, hence
the problem. 

Possible workarounds:
 * Avoid database versions in question
 * change the encoding to {{utf8}}, which is [not recommended however|https://stackoverflow.com/a/766996/6699237]
 * use a MariaDB/MySQL Docker container

 

Solution in Airflow could be to turn on [{{innodb_large_prefix}} and related configs|https://wiki.archlinux.org/index.php/MySQL#Increase_character_limit].
However, this requires the option {{ROW_FORMAT=DYNAMIC}} (and maybe also {{ENGINE=InnoDB}})
to be set on each CREATE statement.

[SqlAlchemy supports this|http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#create-table-arguments-including-storage-engines],
but the question is whether Airflow has some mechanics built in to pass this option in via
some config?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message