From commits-return-23857-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Thu Sep 27 11:14:06 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id B76B0180652 for ; Thu, 27 Sep 2018 11:14:05 +0200 (CEST) Received: (qmail 19635 invoked by uid 500); 27 Sep 2018 09:14:04 -0000 Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list commits@airflow.incubator.apache.org Received: (qmail 19626 invoked by uid 99); 27 Sep 2018 09:14:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Sep 2018 09:14:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 68DA7180675 for ; Thu, 27 Sep 2018 09:14:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -108.751 X-Spam-Level: X-Spam-Status: No, score=-108.751 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, KAM_LOTSOFHASH=0.25, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id NiKJgev0zkcQ for ; Thu, 27 Sep 2018 09:14:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 31B7C5F2F0 for ; Thu, 27 Sep 2018 09:14:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id B20C6E0E5D for ; Thu, 27 Sep 2018 09:14:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 7583D23F9F for ; Thu, 27 Sep 2018 09:14:00 +0000 (UTC) Date: Thu, 27 Sep 2018 09:14:00 +0000 (UTC) From: "Ash Berlin-Taylor (JIRA)" To: commits@airflow.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (AIRFLOW-668) Configuration parsing doesn't work properly with python 3 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AIRFLOW-668?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-668. --------------------------------------- Resolution: Fixed No longer an issue. Tested with this config on Python3.5.2 {code} sql_alchemy_conn_cmd =3D echo sqlite:////Users/ash/airflow/airflow-2.db {code} > Configuration parsing doesn't work properly with python 3 > --------------------------------------------------------- > > Key: AIRFLOW-668 > URL: https://issues.apache.org/jira/browse/AIRFLOW-668 > Project: Apache Airflow > Issue Type: Bug > Environment: Airflow version: v1.7.1.3 > - Airflow components: webserver and scheduler with a postgres database an= d CeleryExecutor > - Python Version: 3.4.5 > Reporter: Rafael Gomes Fernandes > Assignee: Rafael Gomes Fernandes > Priority: Major > > The problem is: if you use python3 and the '_cmd' on the config file airf= low will not start due the error: > {noformat} > File "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configurati= on.py", line 447, in _validate > "sqlite" in self.get('core', 'sql_alchemy_conn')): > TypeError: 'str' does not support the buffer interface > {noformat} > To reproduce the problem change the following line on airflow.cfg: > {code:title=3Dairflow.cfg|borderStyle=3Dsolid} > sql_alchemy_conn_cmd =3D echo sqlite:////~/airflow/airflow.db > {code} > The solution is change the following run_command method's line on airflow= /configuration.py: > {code:title=3Dairflow/configuration.py|borderStyle=3Dsolid} > command.split(), stdout=3Dsubprocess.PIPE, stderr=3Dsubprocess.PI= PE, universal_newlines=3DTrue) > {code} > By setting the universal_newlines to true the file objects stdout and std= err are opened as text files and treated as string in python 2 and python 3= avoiding the error. > run_command with universal_newlines=3DTrue: > When using python 3 output type: and no error. > When using python 2 output type: and no error. > run_command as it is : > When using python 3 output type: and TypeError. > When using python 2 output type: and no error. > I have tested the change with Travis CI and it passed, but when I tried t= o run tox or unit test I found this problems: > {code:title=3Drun_unit_tests.sh|borderStyle=3Dsolid} > ./run_unit_tests.sh -q -c airflow.cfg > Initializing the DB > Starting the unit tests with the following nose arguments: -q -c airflow.= cfg > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > ERROR: Failure: OperationalError ((sqlite3.OperationalError) no such tabl= e: task_instance [SQL: 'DELETE FROM task_instance WHERE task_instance.dag_i= d =3D ?'] [parameters: ('unit_tests',)]) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/f= ailure.py", line 39, in runTest > raise self.exc_val.with_traceback(self.tb) > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/l= oader.py", line 418, in loadTestsFromName > addr.filename, addr.module) > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/i= mporter.py", line 47, in importFromPath > return self.importFromDir(dir_path, fqname) > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/i= mporter.py", line 94, in importFromDir > mod =3D load_module(part_fqname, fh, filename, desc) > File "/home/varrun/test/env/airflow3/lib/python3.4/imp.py", line 245, i= n load_module > return load_package(name, filename) > File "/home/varrun/test/env/airflow3/lib/python3.4/imp.py", line 217, i= n load_package > return methods.load() > File "", line 1220, in load > File "", line 1200, in _load_unlocked > File "", line 1129, in _exec > File "", line 1471, in exec_module > File "", line 321, in _call_with_frames_re= moved > File "/home/varrun/dev/incubator-airflow/tests/__init__.py", line 18, i= n > from .contrib import * > File "/home/varrun/dev/incubator-airflow/tests/contrib/__init__.py", li= ne 16, in > from .operators import * > File "/home/varrun/dev/incubator-airflow/tests/contrib/operators/__init= __.py", line 17, in > from .ssh_execute_operator import * > File "/home/varrun/dev/incubator-airflow/tests/contrib/operators/ssh_ex= ecute_operator.py", line 37, in > reset() > File "/home/varrun/dev/incubator-airflow/tests/contrib/operators/ssh_ex= ecute_operator.py", line 33, in reset > tis.delete() > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalc= hemy/orm/query.py", line 3155, in delete > delete_op.exec_() > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalc= hemy/orm/persistence.py", line 1168, in exec_ > self._do_exec() > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalc= hemy/orm/persistence.py", line 1352, in _do_exec > mapper=3Dself.mapper) > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalc= hemy/orm/session.py", line 1107, in execute > bind, close_with_result=3DTrue).execute(clause, params or {}) > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalc= hemy/engine/base.py", line 945, in execute > return meth(self, multiparams, params) > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalc= hemy/sql/elements.py", line 263, in _execute_on_connection > return connection._execute_clauseelement(self, multiparams, params) > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalc= hemy/engine/base.py", line 1053, in _execute_clauseelement > compiled_sql, distilled_params > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalc= hemy/engine/base.py", line 1189, in _execute_context > context) > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalc= hemy/engine/base.py", line 1393, in _handle_dbapi_exception > exc_info > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalc= hemy/util/compat.py", line 202, in raise_from_cause > reraise(type(exception), exception, tb=3Dexc_tb, cause=3Dcause) > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalc= hemy/util/compat.py", line 185, in reraise > raise value.with_traceback(tb) > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalc= hemy/engine/base.py", line 1182, in _execute_context > context) > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalc= hemy/engine/default.py", line 469, in do_execute > cursor.execute(statement, parameters) > nose.proxy.OperationalError: (sqlite3.OperationalError) no such table: ta= sk_instance [SQL: 'DELETE FROM task_instance WHERE task_instance.dag_id =3D= ?'] [parameters: ('unit_tests',)] > -------------------- >> begin captured logging << -------------------- > root: INFO: Reading the config from /home/varrun/dev/incubator-airflow/ai= rflow.cfg > --------------------- >> end captured logging << --------------------- > ---------------------------------------------------------------------- > Ran 1 test in 0.533s > {code} > {code:title=3Drun_tox.sh|borderStyle=3Dsolid} > ./run_tox.sh > python setup.py test --tox-args=3D"-v -e $TOX_ENV" > gitpython not found: Cannot compute the git version. > running test > Searching for zope.deprecation<5.0,>=3D4.0 > Reading https://pypi.python.org/simple/zope.deprecation/ > Downloading https://pypi.python.org/packages/51/85/b9d2fdcaf38ce3271ad759= c7e8f0e9a28f6d900c1ad495085e4ab1de3795/zope.deprecation-4.2.0.tar.gz#md5=3D= 5d110c34ab7b313490c1a8533bbdc7a0 > Best match: zope.deprecation 4.2.0 > Processing zope.deprecation-4.2.0.tar.gz > Writing /tmp/easy_install-_usl2kjj/zope.deprecation-4.2.0/setup.cfg > Running zope.deprecation-4.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy= _install-_usl2kjj/zope.deprecation-4.2.0/egg-dist-tmp-05ix8s5a > warning: no previously-included files matching '*.pyc' found anywhere in = distribution > creating /home/varrun/dev/incubator-airflow/.eggs/zope.deprecation-4.2.0-= py3.4.egg > Extracting zope.deprecation-4.2.0-py3.4.egg to /home/varrun/dev/incubator= -airflow/.eggs > Installed /home/varrun/dev/incubator-airflow/.eggs/zope.deprecation-4.2.0= -py3.4.egg > Searching for tabulate<0.8.0,>=3D0.7.5 > Reading https://pypi.python.org/simple/tabulate/ > Downloading https://pypi.python.org/packages/1c/a1/3367581782ce79b727954f= 7aa5d29e6a439dc2490a9ac0e7ea0a7115435d/tabulate-0.7.7.tar.gz#md5=3D39a21aaa= 9c10be0749c545be34552559 > Best match: tabulate 0.7.7 > Processing tabulate-0.7.7.tar.gz > Writing /tmp/easy_install-lyp061v6/tabulate-0.7.7/setup.cfg > Running tabulate-0.7.7/setup.py -q bdist_egg --dist-dir /tmp/easy_install= -lyp061v6/tabulate-0.7.7/egg-dist-tmp-3lpm_una > zip_safe flag not set; analyzing archive contents... > Moving tabulate-0.7.7-py3.4.egg to /home/varrun/dev/incubator-airflow/.eg= gs > Installed /home/varrun/dev/incubator-airflow/.eggs/tabulate-0.7.7-py3.4.e= gg > Searching for python-nvd3=3D=3D0.14.2 > Reading https://pypi.python.org/simple/python-nvd3/ > Downloading https://pypi.python.org/packages/30/68/5d5d7c1a46de23f1da3b4a= 7035ac305b76aba582648d19cd9da89b5bd8f6/python-nvd3-0.14.2.tar.gz#md5=3D3a9c= c3024368f2542048227a90459cfa > Best match: python-nvd3 0.14.2 > Processing python-nvd3-0.14.2.tar.gz > Writing /tmp/easy_install-kjjo6ak_/python-nvd3-0.14.2/setup.cfg > Running python-nvd3-0.14.2/setup.py -q bdist_egg --dist-dir /tmp/easy_ins= tall-kjjo6ak_/python-nvd3-0.14.2/egg-dist-tmp-tfryzbr3 > file nvd3.py (for module nvd3) not found > warning: no previously-included files matching '*.pyc' found anywhere in = distribution > warning: no previously-included files matching '__pycache__' found under = directory '*' > warning: no previously-included files matching '*.py[co]' found under dir= ectory '*' > file nvd3.py (for module nvd3) not found > file nvd3.py (for module nvd3) not found > creating /home/varrun/dev/incubator-airflow/.eggs/python_nvd3-0.14.2-py3.= 4.egg > Extracting python_nvd3-0.14.2-py3.4.egg to /home/varrun/dev/incubator-air= flow/.eggs > Installed /home/varrun/dev/incubator-airflow/.eggs/python_nvd3-0.14.2-py3= .4.egg > Searching for psutil<5.0.0,>=3D4.2.0 > Reading https://pypi.python.org/simple/psutil/ > Downloading https://pypi.python.org/packages/6c/49/0f784a247868e167389f6a= c76b8699b2f3d6f4e8e85685dfec43e58d1ed1/psutil-4.4.2.tar.gz#md5=3D189749a7b0= 0813dffd6dac516a1166ef > Best match: psutil 4.4.2 > Processing psutil-4.4.2.tar.gz > Writing /tmp/easy_install-vygvwvy5/psutil-4.4.2/setup.cfg > Running psutil-4.4.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-v= ygvwvy5/psutil-4.4.2/egg-dist-tmp-y3hgclau > warning: no previously-included files matching '*' found under directory = 'docs/_build' > psutil/_psutil_linux.c: In function =E2=80=98PyInit__psutil_linux=E2=80= =99: > psutil/_psutil_linux.c:628: warning: unused variable =E2=80=98v=E2=80=99 > zip_safe flag not set; analyzing archive contents... > psutil.tests.__pycache__.__init__.cpython-34: module references __file__ > psutil.tests.__pycache__.test_linux.cpython-34: module references __file_= _ > psutil.tests.__pycache__.test_misc.cpython-34: module references __file__ > psutil.tests.__pycache__.test_osx.cpython-34: module references __file__ > psutil.tests.__pycache__.runner.cpython-34: module references __file__ > psutil.tests.__pycache__.test_sunos.cpython-34: module references __file_= _ > psutil.tests.__pycache__.test_system.cpython-34: module references __file= __ > psutil.tests.__pycache__.test_posix.cpython-34: module references __file_= _ > psutil.tests.__pycache__.test_windows.cpython-34: module references __fil= e__ > psutil.tests.__pycache__.test_process.cpython-34: module references __fil= e__ > psutil.tests.__pycache__.test_bsd.cpython-34: module references __file__ > psutil.tests.__pycache__.test_memory_leaks.cpython-34: module references = __file__ > psutil.__pycache__.__init__.cpython-34: module references __file__ > psutil.__pycache__._psutil_linux.cpython-34: module references __file__ > psutil.__pycache__._psutil_posix.cpython-34: module references __file__ > creating /home/varrun/dev/incubator-airflow/.eggs/psutil-4.4.2-py3.4-linu= x-x86_64.egg > Extracting psutil-4.4.2-py3.4-linux-x86_64.egg to /home/varrun/dev/incuba= tor-airflow/.eggs > Installed /home/varrun/dev/incubator-airflow/.eggs/psutil-4.4.2-py3.4-lin= ux-x86_64.egg > Searching for lxml<4.0,>=3D3.6.0 > Reading https://pypi.python.org/simple/lxml/ > Downloading https://pypi.python.org/packages/4f/3f/cf6daac551fc36cddafa1a= 71ed48ea5fd642e5feabd3a0d83b8c3dfd0cb4/lxml-3.6.4.tar.gz#md5=3D6dd731423302= 9d9dab0156e7b1c7830b > Best match: lxml 3.6.4 > Processing lxml-3.6.4.tar.gz > Writing /tmp/easy_install-3iiktavg/lxml-3.6.4/setup.cfg > Running lxml-3.6.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-3ii= ktavg/lxml-3.6.4/egg-dist-tmp-9hdcvjnx > Building lxml version 3.6.4. > Building without Cython. > Using build configuration of libxslt 1.1.26 > Building against libxml2/libxslt in the following directory: /usr/lib64 > creating /home/varrun/dev/incubator-airflow/.eggs/lxml-3.6.4-py3.4-linux-= x86_64.egg > Extracting lxml-3.6.4-py3.4-linux-x86_64.egg to /home/varrun/dev/incubato= r-airflow/.eggs > Installed /home/varrun/dev/incubator-airflow/.eggs/lxml-3.6.4-py3.4-linux= -x86_64.egg > Searching for gitpython>=3D2.0.2 > Reading https://pypi.python.org/simple/gitpython/ > Downloading https://pypi.python.org/packages/d7/16/e914d345d7d4e988f9196b= 9719a99220bad6a5dbd636162f9b5cc35f3bd6/GitPython-2.1.0.tar.gz#md5=3D29b1fcf= 504d080dc7a5e630957e829d7 > Best match: GitPython 2.1.0 > Processing GitPython-2.1.0.tar.gz > Writing /tmp/easy_install-bucom7gp/GitPython-2.1.0/setup.cfg > Running GitPython-2.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_instal= l-bucom7gp/GitPython-2.1.0/egg-dist-tmp-_0q9p8l9 > /home/varrun/test/env/airflow3/lib/python3.4/distutils/dist.py:260: UserW= arning: Unknown distribution option: 'test_requirements' > warnings.warn(msg) > warning: no files found matching 'README' > warning: no previously-included files matching '__pycache__' found anywhe= re in distribution > warning: no previously-included files matching '*.pyc' found anywhere in = distribution > creating /home/varrun/dev/incubator-airflow/.eggs/GitPython-2.1.0-py3.4.e= gg > Extracting GitPython-2.1.0-py3.4.egg to /home/varrun/dev/incubator-airflo= w/.eggs > Installed /home/varrun/dev/incubator-airflow/.eggs/GitPython-2.1.0-py3.4.= egg > Traceback (most recent call last): > File "setup.py", line 281, in > do_setup() > File "setup.py", line 275, in do_setup > 'extra_clean': CleanCommand, > File "/home/varrun/test/env/airflow3/lib/python3.4/distutils/core.py", = line 148, in setup > dist.run_commands() > File "/home/varrun/test/env/airflow3/lib/python3.4/distutils/dist.py", = line 955, in run_commands > self.run_command(cmd) > File "/home/varrun/test/env/airflow3/lib/python3.4/distutils/dist.py", = line 974, in run_command > cmd_obj.run() > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setupt= ools-27.2.0-py3.4.egg/setuptools/command/test.py", line 161, in run > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setupt= ools-27.2.0-py3.4.egg/setuptools/dist.py", line 361, in fetch_build_eggs > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setupt= ools-27.2.0-py3.4.egg/pkg_resources/__init__.py", line 851, in resolve > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setupt= ools-27.2.0-py3.4.egg/pkg_resources/__init__.py", line 1116, in best_match > File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setupt= ools-27.2.0-py3.4.egg/pkg_resources/__init__.py", line 720, in find > pkg_resources.VersionConflict: (funcsigs 0.4 (/home/varrun/test/env/airfl= ow3/lib/python3.4/site-packages), Requirement.parse('funcsigs<1.1,>=3D1.0.2= ')) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)