airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rafael Gomes Fernandes (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-668) Configuration parsing doesn't work properly with python 3
Date Fri, 02 Dec 2016 19:51:58 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rafael Gomes Fernandes updated AIRFLOW-668:
-------------------------------------------
    Description: 
The problem is: if you use python3 and the '_cmd' on the config file airflow will not start
due the error:
{noformat}
File "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py", line 447,
in _validate
    "sqlite" in self.get('core', 'sql_alchemy_conn')):
TypeError: 'str' does not support the buffer interface
{noformat}

To reproduce the problem change the following line on airflow.cfg:
{code:title=airflow.cfg|borderStyle=solid}
sql_alchemy_conn_cmd = echo sqlite:////~/airflow/airflow.db
{code}

The solution is change the following run_command method's line on airflow/configuration.py:
{code:title=airflow/configuration.py|borderStyle=solid}
        command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
{code}

By setting the universal_newlines to true the file objects stdout and stderr are opened as
text files and treated as string in python 2 and python 3 avoiding the error.

run_command with universal_newlines=True:

When using python 3 output type: <class 'str'> and no error.

When using python 2 output type: <type 'str'> and no error.


run_command as it is :
When using python 3 output type: <class 'bytes'> and TypeError.

When using python 2 output type: <type 'str'> and no error.


  was:
The problem is: if you use python3 and the '_cmd' on the config file airflow will not start
due the error:
{noformat}
File "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py", line 447,
in _validate
    "sqlite" in self.get('core', 'sql_alchemy_conn')):
TypeError: 'str' does not support the buffer interface
{noformat}

To reproduce the problem change the following line on airflow.cfg:
{code:title=airflow.cfg|borderStyle=solid}
sql_alchemy_conn_cmd = echo sqlite:////~/airflow/airflow.db
{code}

The solution is change the following run_command method's line on airflow/configuration.py:
{code:title=airflow/configuration.py|borderStyle=solid}
{color:green} 
        command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
{color}
{code}

By setting the universal_newlines to true the file objects stdout and stderr are opened as
text files and treated as string in python 2 and python 3 avoiding the error.

run_command with universal_newlines=True:

When using python 3 output type: <class 'str'> and no error.

When using python 2 output type: <type 'str'> and no error.


run_command as it is :
When using python 3 output type: <class 'bytes'> and TypeError.

When using python 2 output type: <type 'str'> and no error.



> Configuration parsing doesn't work properly with python 3
> ---------------------------------------------------------
>
>                 Key: AIRFLOW-668
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-668
>             Project: Apache Airflow
>          Issue Type: Bug
>         Environment:  Airflow version: v1.7.1.3
> - Airflow components: webserver and scheduler with a postgres database and CeleryExecutor
> - Python Version: 3.4.5
>            Reporter: Rafael Gomes Fernandes
>            Assignee: Rafael Gomes Fernandes
>
> The problem is: if you use python3 and the '_cmd' on the config file airflow will not
start due the error:
> {noformat}
> File "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py", line
447, in _validate
>     "sqlite" in self.get('core', 'sql_alchemy_conn')):
> TypeError: 'str' does not support the buffer interface
> {noformat}
> To reproduce the problem change the following line on airflow.cfg:
> {code:title=airflow.cfg|borderStyle=solid}
> sql_alchemy_conn_cmd = echo sqlite:////~/airflow/airflow.db
> {code}
> The solution is change the following run_command method's line on airflow/configuration.py:
> {code:title=airflow/configuration.py|borderStyle=solid}
>         command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
> {code}
> By setting the universal_newlines to true the file objects stdout and stderr are opened
as text files and treated as string in python 2 and python 3 avoiding the error.
> run_command with universal_newlines=True:
> When using python 3 output type: <class 'str'> and no error.
> When using python 2 output type: <type 'str'> and no error.
> run_command as it is :
> When using python 3 output type: <class 'bytes'> and TypeError.
> When using python 2 output type: <type 'str'> and no error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message