airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Koeltringer <>
Subject MariaDB max key length issue
Date Tue, 06 Feb 2018 11:07:17 GMT

I was trying to switch from Sqlite/SequentialExecutor over to 

However, the ``airflow initdb`` step failed, with the message: 
``Specified key was too long; max key length is 767 bytes``

Turns out that the default charset in MariaDB is ``utf8mb4``, and the 
default max. keylength (**) is 767 bytes. The field ``dag_id`` is 
defined as ``VARCHAR(250)``, 250 x 4 = 1000 > 767, hence the problem.

There are three solutions (workarounds) to this:

change the encoding to ``utf8``, which is not recommended however [3]

use a recent MariaDB/MySQL version via Docker, which is fine on my 
machine, but cannot be used in production.

turn on ``innodb_large_prefix`` and related configs [4]. However, this 
requires the option "ROW_FORMAT=DYNAMIC" (and maybe also 
"ENGINE=InnoDB") to be set on each CREATE statement.

SqlAlchemy supports this [5], but the question is whether Airflow has 
some mechanics built in to pass this option in via some config? Or did I 
miss something / is there a fix already?

BTW: I am using MariaDB v10.1.30 (provided by my system)

Thanks in advance,

(**) MariaDB < 10.2.2 [1] // MySQL < 5.7.7 [2]






Andreas Koeltringer

View raw message