From commits-return-74616-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Sat Nov 9 10:53:06 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 281081804BB for ; Sat, 9 Nov 2019 11:53:06 +0100 (CET) Received: (qmail 11489 invoked by uid 500); 9 Nov 2019 10:53:05 -0000 Mailing-List: contact commits-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list commits@airflow.apache.org Received: (qmail 11480 invoked by uid 99); 9 Nov 2019 10:53:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Nov 2019 10:53:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id BB453C279E for ; Sat, 9 Nov 2019 10:53:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -112.25 X-Spam-Level: X-Spam-Status: No, score=-112.25 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_INFOUSMEBIZ=0.75, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id FinRopNPNW5W for ; Sat, 9 Nov 2019 10:53:02 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=207.244.88.153; helo=mail.apache.org; envelope-from=jira@apache.org; receiver= Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with SMTP id 4F656BC509 for ; Sat, 9 Nov 2019 10:53:02 +0000 (UTC) Received: (qmail 11185 invoked by uid 99); 9 Nov 2019 10:53:02 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Nov 2019 10:53:02 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 2C6D3E0F04 for ; Sat, 9 Nov 2019 10:53:00 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 182DC7802C3 for ; Sat, 9 Nov 2019 10:53:00 +0000 (UTC) Date: Sat, 9 Nov 2019 10:53:00 +0000 (UTC) From: "Sayed Mohammad Hossein Torabi (Jira)" To: commits@airflow.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (AIRFLOW-5884) Airflow cant create migration on MySQL MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Sayed Mohammad Hossein Torabi created AIRFLOW-5884: ------------------------------------------------------ Summary: Airflow cant create migration on MySQL Key: AIRFLOW-5884 URL: https://issues.apache.org/jira/browse/AIRFLOW-5884 Project: Apache Airflow Issue Type: Bug Components: cli, database Affects Versions: 1.10.6, 1.10.5 Environment: python3.7 10.1.38-MariaDB-0+deb9u1 Debian 9.8 Reporter: Sayed Mohammad Hossein Torabi Airflow can't set up with *Mysql*. Here is the log of *airflow db initdb*: =C2=A0 {code:java} // 2019-11-09 13:57:48,695] {settings.py:213} INFO - settings.configure_orm= (): Using pool settings. pool_size=3D5, max_overflow=3D10, pool_recycle=3D1= 800, pid=3D59502019-11-09 13:57:48,695] {settings.py:213} INFO - settings.c= onfigure_orm(): Using pool settings. pool_size=3D5, max_overflow=3D10, pool= _recycle=3D1800, pid=3D5950[2019-11-09 13:57:49,237] {default_celery.py:90}= WARNING - You have configured a result_backend of pyamqp://admin:Al0peykHa= doop@172.16.2.230:5672/airflow, it is highly recommended to use an alternat= ive result_backend (i.e. a database).[2019-11-09 13:57:49,239] {__init__.py= :51} INFO - Using executor CeleryExecutorDB: mysql://root:***@172.16.2.230:= 3306/airflowThis will drop existing tables if they exist. Proceed? (y/n)y[2= 019-11-09 13:57:50,627] {db.py:390} INFO - Dropping tables that exist[2019-= 11-09 13:57:50,679] {migration.py:130} INFO - Context impl MySQLImpl.[2019-= 11-09 13:57:50,680] {migration.py:137} INFO - Will assume non-transactional= DDL.[2019-11-09 13:57:50,691] {db.py:369} INFO - Creating tablesINFO=C2=A0= [alembic.runtime.migration] Context impl MySQLImpl.INFO=C2=A0 [alembic.run= time.migration] Will assume non-transactional DDL.INFO=C2=A0 [alembic.runti= me.migration] Running upgrade=C2=A0 -> e3a246e0dc1, current schemaTraceback= (most recent call last):=C2=A0 File "/usr/local/lib/python3.7/site-package= s/sqlalchemy/engine/base.py", line 1244, in _execute_context=C2=A0 =C2=A0 c= ursor, statement, parameters, context=C2=A0 File "/usr/local/lib/python3.7/= site-packages/sqlalchemy/engine/default.py", line 552, in do_execute=C2=A0 = =C2=A0 cursor.execute(statement, parameters)=C2=A0 File "/usr/local/lib/pyt= hon3.7/site-packages/MySQLdb/cursors.py", line 255, in execute=C2=A0 =C2=A0= self.errorhandler(self, exc, value)=C2=A0 File "/usr/local/lib/python3.7/s= ite-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler=C2=A0= =C2=A0 raise errorvalue=C2=A0 File "/usr/local/lib/python3.7/site-packages= /MySQLdb/cursors.py", line 252, in execute=C2=A0 =C2=A0 res =3D self._query= (query)=C2=A0 File "/usr/local/lib/python3.7/site-packages/MySQLdb/cursors.= py", line 378, in _query=C2=A0 =C2=A0 db.query(q)=C2=A0 File "/usr/local/li= b/python3.7/site-packages/MySQLdb/connections.py", line 280, in query=C2=A0= =C2=A0 _mysql.connection.query(self, query)_mysql_exceptions.OperationalEr= ror: (1071, 'Specified key was too long; max key length is 767 bytes') The above exception was the direct cause of the following exception: Traceback (most recent call last):=C2=A0 File "/usr/local/bin/airflow", lin= e 32, in =C2=A0 =C2=A0 args.func(args)=C2=A0 File "/usr/local/lib/p= ython3.7/site-packages/airflow/bin/cli.py", line 1112, in resetdb=C2=A0 =C2= =A0 db.resetdb(settings.RBAC)=C2=A0 File "/usr/local/lib/python3.7/site-pac= kages/airflow/utils/db.py", line 406, in resetdb=C2=A0 =C2=A0 initdb(rbac)= =C2=A0 File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", l= ine 106, in initdb=C2=A0 =C2=A0 upgradedb()=C2=A0 File "/usr/local/lib/pyth= on3.7/site-packages/airflow/utils/db.py", line 377, in upgradedb=C2=A0 =C2= =A0 command.upgrade(config, 'heads')=C2=A0 File "/usr/local/lib/python3.7/s= ite-packages/alembic/command.py", line 276, in upgrade=C2=A0 =C2=A0 script.= run_env()=C2=A0 File "/usr/local/lib/python3.7/site-packages/alembic/script= /base.py", line 475, in run_env=C2=A0 =C2=A0 util.load_python_file(self.dir= , "env.py")=C2=A0 File "/usr/local/lib/python3.7/site-packages/alembic/util= /pyfiles.py", line 90, in load_python_file=C2=A0 =C2=A0 module =3D load_mod= ule_py(module_id, path)=C2=A0 File "/usr/local/lib/python3.7/site-packages/= alembic/util/compat.py", line 177, in load_module_py=C2=A0 =C2=A0 spec.load= er.exec_module(module)=C2=A0 File "",= line 728, in exec_module=C2=A0 File "", line = 219, in _call_with_frames_removed=C2=A0 File "/usr/local/lib/python3.7/site= -packages/airflow/migrations/env.py", line 92, in =C2=A0 =C2=A0 run= _migrations_online()=C2=A0 File "/usr/local/lib/python3.7/site-packages/air= flow/migrations/env.py", line 86, in run_migrations_online=C2=A0 =C2=A0 con= text.run_migrations()=C2=A0 File "", line 8, in run_migrations=C2= =A0 File "/usr/local/lib/python3.7/site-packages/alembic/runtime/environmen= t.py", line 839, in run_migrations=C2=A0 =C2=A0 self.get_context().run_migr= ations(**kw)=C2=A0 File "/usr/local/lib/python3.7/site-packages/alembic/run= time/migration.py", line 362, in run_migrations=C2=A0 =C2=A0 step.migration= _fn(**kw)=C2=A0 File "/usr/local/lib/python3.7/site-packages/airflow/migrat= ions/versions/e3a246e0dc1_current_schema.py", line 73, in upgrade=C2=A0 =C2= =A0 sa.PrimaryKeyConstraint('dag_id')=C2=A0 File "", line 8, in cre= ate_table=C2=A0 File "", line 3, in create_table=C2=A0 File "/usr/l= ocal/lib/python3.7/site-packages/alembic/operations/ops.py", line 1248, in = create_table=C2=A0 =C2=A0 return operations.invoke(op)=C2=A0 File "/usr/loc= al/lib/python3.7/site-packages/alembic/operations/base.py", line 345, in in= voke=C2=A0 =C2=A0 return fn(self, operation)=C2=A0 File "/usr/local/lib/pyt= hon3.7/site-packages/alembic/operations/toimpl.py", line 101, in create_tab= le=C2=A0 =C2=A0 operations.impl.create_table(table)=C2=A0 File "/usr/local/= lib/python3.7/site-packages/alembic/ddl/impl.py", line 252, in create_table= =C2=A0 =C2=A0 self._exec(schema.CreateTable(table))=C2=A0 File "/usr/local/= lib/python3.7/site-packages/alembic/ddl/impl.py", line 134, in _exec=C2=A0 = =C2=A0 return conn.execute(construct, *multiparams, **params)=C2=A0 File "/= usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 988,= in execute=C2=A0 =C2=A0 return meth(self, multiparams, params)=C2=A0 File = "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 72, in= _execute_on_connection=C2=A0 =C2=A0 return connection._execute_ddl(self, m= ultiparams, params)=C2=A0 File "/usr/local/lib/python3.7/site-packages/sqla= lchemy/engine/base.py", line 1050, in _execute_ddl=C2=A0 =C2=A0 compiled,= =C2=A0 File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.= py", line 1248, in _execute_context=C2=A0 =C2=A0 e, statement, parameters, = cursor, context=C2=A0 File "/usr/local/lib/python3.7/site-packages/sqlalche= my/engine/base.py", line 1466, in _handle_dbapi_exception=C2=A0 =C2=A0 util= .raise_from_cause(sqlalchemy_exception, exc_info)=C2=A0 File "/usr/local/li= b/python3.7/site-packages/sqlalchemy/util/compat.py", line 398, in raise_fr= om_cause=C2=A0 =C2=A0 reraise(type(exception), exception, tb=3Dexc_tb, caus= e=3Dcause)=C2=A0 File "/usr/local/lib/python3.7/site-packages/sqlalchemy/ut= il/compat.py", line 152, in reraise=C2=A0 =C2=A0 raise value.with_traceback= (tb)=C2=A0 File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/b= ase.py", line 1244, in _execute_context=C2=A0 =C2=A0 cursor, statement, par= ameters, context=C2=A0 File "/usr/local/lib/python3.7/site-packages/sqlalch= emy/engine/default.py", line 552, in do_execute=C2=A0 =C2=A0 cursor.execute= (statement, parameters)=C2=A0 File "/usr/local/lib/python3.7/site-packages/= MySQLdb/cursors.py", line 255, in execute=C2=A0 =C2=A0 self.errorhandler(se= lf, exc, value)=C2=A0 File "/usr/local/lib/python3.7/site-packages/MySQLdb/= connections.py", line 50, in defaulterrorhandler=C2=A0 =C2=A0 raise errorva= lue=C2=A0 File "/usr/local/lib/python3.7/site-packages/MySQLdb/cursors.py",= line 252, in execute=C2=A0 =C2=A0 res =3D self._query(query)=C2=A0 File "/= usr/local/lib/python3.7/site-packages/MySQLdb/cursors.py", line 378, in _qu= ery=C2=A0 =C2=A0 db.query(q)=C2=A0 File "/usr/local/lib/python3.7/site-pack= ages/MySQLdb/connections.py", line 280, in query=C2=A0 =C2=A0 _mysql.connec= tion.query(self, query)sqlalchemy.exc.OperationalError: (_mysql_exceptions.= OperationalError) (1071, 'Specified key was too long; max key length is 767= bytes')[SQL:=C2=A0CREATE TABLE dag ( dag_id VARCHAR(250) NOT NULL,=C2=A0 i= s_paused BOOL,=C2=A0 is_subdag BOOL,=C2=A0 is_active BOOL,=C2=A0 last_sched= uler_run DATETIME,=C2=A0 last_pickled DATETIME,=C2=A0 last_expired DATETIME= ,=C2=A0 scheduler_lock BOOL,=C2=A0 pickle_id INTEGER,=C2=A0 fileloc VARCHAR= (2000),=C2=A0 owners VARCHAR(2000),=C2=A0 PRIMARY KEY (dag_id),=C2=A0 CHECK= (is_paused IN (0, 1)),=C2=A0 CHECK (is_subdag IN (0, 1)),=C2=A0 CHECK (is_= active IN (0, 1)),=C2=A0 CHECK (scheduler_lock IN (0, 1))) ](Background on this error at: http://sqlalche.me/e/e3q8) {code} It seems this issue related to the *dag table* because mysql can't create v= archar column more than 767 bytes. I tried to create this table on MySQL an= d the results were the same. here is the result: =C2=A0 =C2=A0 {code:java} //MariaDB [airflow]> CREATE TABLE dag ( -> dag_id VARCHAR(250) NOT NULL,=20 -> is_paused BOOL,=20 -> is_subdag BOOL,=20 -> is_active BOOL,=20 -> last_scheduler_run DATETIME,=20 -> last_pickled DATETIME,=20 -> last_expired DATETIME,=20 -> scheduler_lock BOOL,=20 -> pickle_id INTEGER,=20 -> fileloc VARCHAR(2000),=20 -> owners VARCHAR(2000),=20 -> PRIMARY KEY (dag_id),=20 -> CHECK (is_paused IN (0, 1)),=20 -> CHECK (is_subdag IN (0, 1)),=20 -> CHECK (is_active IN (0, 1)),=20 -> CHECK (scheduler_lock IN (0, 1)) -> ) -> ; ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes {code} =C2=A0 =C2=A0 -- This message was sent by Atlassian Jira (v8.3.4#803005)