airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rafael Gomes Fernandes (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-668) Configuration parsing doesn't work properly with python 3
Date Fri, 02 Dec 2016 21:30:58 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rafael Gomes Fernandes updated AIRFLOW-668:
-------------------------------------------
    External issue URL: https://github.com/apache/incubator-airflow/pull/1918

> Configuration parsing doesn't work properly with python 3
> ---------------------------------------------------------
>
>                 Key: AIRFLOW-668
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-668
>             Project: Apache Airflow
>          Issue Type: Bug
>         Environment:  Airflow version: v1.7.1.3
> - Airflow components: webserver and scheduler with a postgres database and CeleryExecutor
> - Python Version: 3.4.5
>            Reporter: Rafael Gomes Fernandes
>            Assignee: Rafael Gomes Fernandes
>
> The problem is: if you use python3 and the '_cmd' on the config file airflow will not
start due the error:
> {noformat}
> File "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py", line
447, in _validate
>     "sqlite" in self.get('core', 'sql_alchemy_conn')):
> TypeError: 'str' does not support the buffer interface
> {noformat}
> To reproduce the problem change the following line on airflow.cfg:
> {code:title=airflow.cfg|borderStyle=solid}
> sql_alchemy_conn_cmd = echo sqlite:////~/airflow/airflow.db
> {code}
> The solution is change the following run_command method's line on airflow/configuration.py:
> {code:title=airflow/configuration.py|borderStyle=solid}
>         command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
> {code}
> By setting the universal_newlines to true the file objects stdout and stderr are opened
as text files and treated as string in python 2 and python 3 avoiding the error.
> run_command with universal_newlines=True:
> When using python 3 output type: <class 'str'> and no error.
> When using python 2 output type: <type 'str'> and no error.
> run_command as it is :
> When using python 3 output type: <class 'bytes'> and TypeError.
> When using python 2 output type: <type 'str'> and no error.
> I have tested the change with Travis CI and it passed, but when I tried to run tox or
unit test I found this problems:
> {code:title=run_unit_tests.sh|borderStyle=solid}
> ./run_unit_tests.sh -q -c airflow.cfg
> Initializing the DB
> Starting the unit tests with the following nose arguments: -q -c airflow.cfg
> ======================================================================
> ERROR: Failure: OperationalError ((sqlite3.OperationalError) no such table: task_instance
[SQL: 'DELETE FROM task_instance WHERE task_instance.dag_id = ?'] [parameters: ('unit_tests',)])
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/failure.py",
line 39, in runTest
>     raise self.exc_val.with_traceback(self.tb)
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/loader.py", line
418, in loadTestsFromName
>     addr.filename, addr.module)
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/importer.py",
line 47, in importFromPath
>     return self.importFromDir(dir_path, fqname)
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/importer.py",
line 94, in importFromDir
>     mod = load_module(part_fqname, fh, filename, desc)
>   File "/home/varrun/test/env/airflow3/lib/python3.4/imp.py", line 245, in load_module
>     return load_package(name, filename)
>   File "/home/varrun/test/env/airflow3/lib/python3.4/imp.py", line 217, in load_package
>     return methods.load()
>   File "<frozen importlib._bootstrap>", line 1220, in load
>   File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
>   File "<frozen importlib._bootstrap>", line 1129, in _exec
>   File "<frozen importlib._bootstrap>", line 1471, in exec_module
>   File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
>   File "/home/varrun/dev/incubator-airflow/tests/__init__.py", line 18, in <module>
>     from .contrib import *
>   File "/home/varrun/dev/incubator-airflow/tests/contrib/__init__.py", line 16, in <module>
>     from .operators import *
>   File "/home/varrun/dev/incubator-airflow/tests/contrib/operators/__init__.py", line
17, in <module>
>     from .ssh_execute_operator import *
>   File "/home/varrun/dev/incubator-airflow/tests/contrib/operators/ssh_execute_operator.py",
line 37, in <module>
>     reset()
>   File "/home/varrun/dev/incubator-airflow/tests/contrib/operators/ssh_execute_operator.py",
line 33, in reset
>     tis.delete()
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/orm/query.py",
line 3155, in delete
>     delete_op.exec_()
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/orm/persistence.py",
line 1168, in exec_
>     self._do_exec()
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/orm/persistence.py",
line 1352, in _do_exec
>     mapper=self.mapper)
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/orm/session.py",
line 1107, in execute
>     bind, close_with_result=True).execute(clause, params or {})
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 945, in execute
>     return meth(self, multiparams, params)
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/sql/elements.py",
line 263, in _execute_on_connection
>     return connection._execute_clauseelement(self, multiparams, params)
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 1053, in _execute_clauseelement
>     compiled_sql, distilled_params
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 1189, in _execute_context
>     context)
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 1393, in _handle_dbapi_exception
>     exc_info
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 202, in raise_from_cause
>     reraise(type(exception), exception, tb=exc_tb, cause=cause)
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/util/compat.py",
line 185, in reraise
>     raise value.with_traceback(tb)
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py",
line 1182, in _execute_context
>     context)
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/default.py",
line 469, in do_execute
>     cursor.execute(statement, parameters)
> nose.proxy.OperationalError: (sqlite3.OperationalError) no such table: task_instance
[SQL: 'DELETE FROM task_instance WHERE task_instance.dag_id = ?'] [parameters: ('unit_tests',)]
> -------------------- >> begin captured logging << --------------------
> root: INFO: Reading the config from /home/varrun/dev/incubator-airflow/airflow.cfg
> --------------------- >> end captured logging << ---------------------
> ----------------------------------------------------------------------
> Ran 1 test in 0.533s
> {code}
> {code:title=run_tox.sh|borderStyle=solid}
> ./run_tox.sh
> python setup.py test --tox-args="-v -e $TOX_ENV"
> gitpython not found: Cannot compute the git version.
> running test
> Searching for zope.deprecation<5.0,>=4.0
> Reading https://pypi.python.org/simple/zope.deprecation/
> Downloading https://pypi.python.org/packages/51/85/b9d2fdcaf38ce3271ad759c7e8f0e9a28f6d900c1ad495085e4ab1de3795/zope.deprecation-4.2.0.tar.gz#md5=5d110c34ab7b313490c1a8533bbdc7a0
> Best match: zope.deprecation 4.2.0
> Processing zope.deprecation-4.2.0.tar.gz
> Writing /tmp/easy_install-_usl2kjj/zope.deprecation-4.2.0/setup.cfg
> Running zope.deprecation-4.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-_usl2kjj/zope.deprecation-4.2.0/egg-dist-tmp-05ix8s5a
> warning: no previously-included files matching '*.pyc' found anywhere in distribution
> creating /home/varrun/dev/incubator-airflow/.eggs/zope.deprecation-4.2.0-py3.4.egg
> Extracting zope.deprecation-4.2.0-py3.4.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/zope.deprecation-4.2.0-py3.4.egg
> Searching for tabulate<0.8.0,>=0.7.5
> Reading https://pypi.python.org/simple/tabulate/
> Downloading https://pypi.python.org/packages/1c/a1/3367581782ce79b727954f7aa5d29e6a439dc2490a9ac0e7ea0a7115435d/tabulate-0.7.7.tar.gz#md5=39a21aaa9c10be0749c545be34552559
> Best match: tabulate 0.7.7
> Processing tabulate-0.7.7.tar.gz
> Writing /tmp/easy_install-lyp061v6/tabulate-0.7.7/setup.cfg
> Running tabulate-0.7.7/setup.py -q bdist_egg --dist-dir /tmp/easy_install-lyp061v6/tabulate-0.7.7/egg-dist-tmp-3lpm_una
> zip_safe flag not set; analyzing archive contents...
> Moving tabulate-0.7.7-py3.4.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/tabulate-0.7.7-py3.4.egg
> Searching for python-nvd3==0.14.2
> Reading https://pypi.python.org/simple/python-nvd3/
> Downloading https://pypi.python.org/packages/30/68/5d5d7c1a46de23f1da3b4a7035ac305b76aba582648d19cd9da89b5bd8f6/python-nvd3-0.14.2.tar.gz#md5=3a9cc3024368f2542048227a90459cfa
> Best match: python-nvd3 0.14.2
> Processing python-nvd3-0.14.2.tar.gz
> Writing /tmp/easy_install-kjjo6ak_/python-nvd3-0.14.2/setup.cfg
> Running python-nvd3-0.14.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-kjjo6ak_/python-nvd3-0.14.2/egg-dist-tmp-tfryzbr3
> file nvd3.py (for module nvd3) not found
> warning: no previously-included files matching '*.pyc' found anywhere in distribution
> warning: no previously-included files matching '__pycache__' found under directory '*'
> warning: no previously-included files matching '*.py[co]' found under directory '*'
> file nvd3.py (for module nvd3) not found
> file nvd3.py (for module nvd3) not found
> creating /home/varrun/dev/incubator-airflow/.eggs/python_nvd3-0.14.2-py3.4.egg
> Extracting python_nvd3-0.14.2-py3.4.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/python_nvd3-0.14.2-py3.4.egg
> Searching for psutil<5.0.0,>=4.2.0
> Reading https://pypi.python.org/simple/psutil/
> Downloading https://pypi.python.org/packages/6c/49/0f784a247868e167389f6ac76b8699b2f3d6f4e8e85685dfec43e58d1ed1/psutil-4.4.2.tar.gz#md5=189749a7b00813dffd6dac516a1166ef
> Best match: psutil 4.4.2
> Processing psutil-4.4.2.tar.gz
> Writing /tmp/easy_install-vygvwvy5/psutil-4.4.2/setup.cfg
> Running psutil-4.4.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-vygvwvy5/psutil-4.4.2/egg-dist-tmp-y3hgclau
> warning: no previously-included files matching '*' found under directory 'docs/_build'
> psutil/_psutil_linux.c: In function ‘PyInit__psutil_linux’:
> psutil/_psutil_linux.c:628: warning: unused variable ‘v’
> zip_safe flag not set; analyzing archive contents...
> psutil.tests.__pycache__.__init__.cpython-34: module references __file__
> psutil.tests.__pycache__.test_linux.cpython-34: module references __file__
> psutil.tests.__pycache__.test_misc.cpython-34: module references __file__
> psutil.tests.__pycache__.test_osx.cpython-34: module references __file__
> psutil.tests.__pycache__.runner.cpython-34: module references __file__
> psutil.tests.__pycache__.test_sunos.cpython-34: module references __file__
> psutil.tests.__pycache__.test_system.cpython-34: module references __file__
> psutil.tests.__pycache__.test_posix.cpython-34: module references __file__
> psutil.tests.__pycache__.test_windows.cpython-34: module references __file__
> psutil.tests.__pycache__.test_process.cpython-34: module references __file__
> psutil.tests.__pycache__.test_bsd.cpython-34: module references __file__
> psutil.tests.__pycache__.test_memory_leaks.cpython-34: module references __file__
> psutil.__pycache__.__init__.cpython-34: module references __file__
> psutil.__pycache__._psutil_linux.cpython-34: module references __file__
> psutil.__pycache__._psutil_posix.cpython-34: module references __file__
> creating /home/varrun/dev/incubator-airflow/.eggs/psutil-4.4.2-py3.4-linux-x86_64.egg
> Extracting psutil-4.4.2-py3.4-linux-x86_64.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/psutil-4.4.2-py3.4-linux-x86_64.egg
> Searching for lxml<4.0,>=3.6.0
> Reading https://pypi.python.org/simple/lxml/
> Downloading https://pypi.python.org/packages/4f/3f/cf6daac551fc36cddafa1a71ed48ea5fd642e5feabd3a0d83b8c3dfd0cb4/lxml-3.6.4.tar.gz#md5=6dd7314233029d9dab0156e7b1c7830b
> Best match: lxml 3.6.4
> Processing lxml-3.6.4.tar.gz
> Writing /tmp/easy_install-3iiktavg/lxml-3.6.4/setup.cfg
> Running lxml-3.6.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-3iiktavg/lxml-3.6.4/egg-dist-tmp-9hdcvjnx
> Building lxml version 3.6.4.
> Building without Cython.
> Using build configuration of libxslt 1.1.26
> Building against libxml2/libxslt in the following directory: /usr/lib64
> creating /home/varrun/dev/incubator-airflow/.eggs/lxml-3.6.4-py3.4-linux-x86_64.egg
> Extracting lxml-3.6.4-py3.4-linux-x86_64.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/lxml-3.6.4-py3.4-linux-x86_64.egg
> Searching for gitpython>=2.0.2
> Reading https://pypi.python.org/simple/gitpython/
> Downloading https://pypi.python.org/packages/d7/16/e914d345d7d4e988f9196b9719a99220bad6a5dbd636162f9b5cc35f3bd6/GitPython-2.1.0.tar.gz#md5=29b1fcf504d080dc7a5e630957e829d7
> Best match: GitPython 2.1.0
> Processing GitPython-2.1.0.tar.gz
> Writing /tmp/easy_install-bucom7gp/GitPython-2.1.0/setup.cfg
> Running GitPython-2.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-bucom7gp/GitPython-2.1.0/egg-dist-tmp-_0q9p8l9
> /home/varrun/test/env/airflow3/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown
distribution option: 'test_requirements'
>   warnings.warn(msg)
> warning: no files found matching 'README'
> warning: no previously-included files matching '__pycache__' found anywhere in distribution
> warning: no previously-included files matching '*.pyc' found anywhere in distribution
> creating /home/varrun/dev/incubator-airflow/.eggs/GitPython-2.1.0-py3.4.egg
> Extracting GitPython-2.1.0-py3.4.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/GitPython-2.1.0-py3.4.egg
> Traceback (most recent call last):
>   File "setup.py", line 281, in <module>
>     do_setup()
>   File "setup.py", line 275, in do_setup
>     'extra_clean': CleanCommand,
>   File "/home/varrun/test/env/airflow3/lib/python3.4/distutils/core.py", line 148, in
setup
>     dist.run_commands()
>   File "/home/varrun/test/env/airflow3/lib/python3.4/distutils/dist.py", line 955, in
run_commands
>     self.run_command(cmd)
>   File "/home/varrun/test/env/airflow3/lib/python3.4/distutils/dist.py", line 974, in
run_command
>     cmd_obj.run()
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/setuptools/command/test.py",
line 161, in run
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/setuptools/dist.py",
line 361, in fetch_build_eggs
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/pkg_resources/__init__.py",
line 851, in resolve
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/pkg_resources/__init__.py",
line 1116, in best_match
>   File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/pkg_resources/__init__.py",
line 720, in find
> pkg_resources.VersionConflict: (funcsigs 0.4 (/home/varrun/test/env/airflow3/lib/python3.4/site-packages),
Requirement.parse('funcsigs<1.1,>=1.0.2'))
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message