airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Koeltringer <andreas.koeltrin...@n-fuse.co>
Subject MariaDB max key length issue
Date Tue, 06 Feb 2018 11:07:17 GMT
Hi,

I was trying to switch from Sqlite/SequentialExecutor over to 
MariaDB/LocalExecutor.

However, the ``airflow initdb`` step failed, with the message: 
``Specified key was too long; max key length is 767 bytes``

Turns out that the default charset in MariaDB is ``utf8mb4``, and the 
default max. keylength (**) is 767 bytes. The field ``dag_id`` is 
defined as ``VARCHAR(250)``, 250 x 4 = 1000 > 767, hence the problem.

There are three solutions (workarounds) to this:

(1)
change the encoding to ``utf8``, which is not recommended however [3]

(2)
use a recent MariaDB/MySQL version via Docker, which is fine on my 
machine, but cannot be used in production.

(3)
turn on ``innodb_large_prefix`` and related configs [4]. However, this 
requires the option "ROW_FORMAT=DYNAMIC" (and maybe also 
"ENGINE=InnoDB") to be set on each CREATE statement.

SqlAlchemy supports this [5], but the question is whether Airflow has 
some mechanics built in to pass this option in via some config? Or did I 
miss something / is there a fix already?


BTW: I am using MariaDB v10.1.30 (provided by my system)


Thanks in advance,
Andreas



(**) MariaDB < 10.2.2 [1] // MySQL < 5.7.7 [2]


[1] 
https://mariadb.com/kb/en/library/xtradbinnodb-server-system-variables/#innodb_large_prefix

[2] 
https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_large_prefix

[3] https://stackoverflow.com/a/766996/6699237

[4] https://wiki.archlinux.org/index.php/MySQL#Increase_character_limit

[5] 
http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#create-table-arguments-including-storage-engines


-- 
Andreas Koeltringer
Mail:   andreas.koeltringer@n-fuse.co

Mime
View raw message