Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
Date: Fri, 2 Dec 2016 19:45:58 +0000 (UTC)
From: "Rafael Gomes Fernandes (JIRA)" <jira@apache.org>
To: commits@airflow.incubator.apache.org
Message-ID: <JIRA.13025181.1480707243000.426684.1480707958741@Atlassian.JIRA>
In-Reply-To: <JIRA.13025181.1480707243000@Atlassian.JIRA>
References: <JIRA.13025181.1480707243000@Atlassian.JIRA> <JIRA.13025181.1480707243186@arcas>
Subject: [jira] [Updated] (AIRFLOW-668) Configuration parsing doesn't work
 properly with python 3
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 02 Dec 2016 19:46:05 -0000


     [ https://issues.apache.org/jira/browse/AIRFLOW-668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rafael Gomes Fernandes updated AIRFLOW-668:
-------------------------------------------
    Description: 
The problem is: if you use python3 and the '_cmd' on the config file airflow will not start due the error:
{noformat}
File "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py", line 447, in _validate
    "sqlite" in self.get('core', 'sql_alchemy_conn')):
TypeError: 'str' does not support the buffer interface
{noformat}

To reproduce the problem change the following line on airflow.cfg:
{code:title=airflow.cfgborderStyle=solid}
sql_alchemy_conn_cmd = echo sqlite:////~/airflow/airflow.db
{code}

The solution is change the following run_command method's line on configuration.py:
{code:title=airflow/configuration.py|borderStyle=solid}
def run_command(command):
    """
    Runs command and returns stdout
    """
    process = subprocess.Popen(
        command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE{color:green}, universal_newlines=True{color})
    output, stderr = process.communicate()

    if process.returncode != 0:
        raise AirflowConfigException(
            "Cannot execute {}. Error code is: {}. Output: {}, Stderr: {}"
            .format(command, process.returncode, output, stderr)
        )

    return output
{code}

By setting the universal_newlines to true the file objects stdout and stderr are opened as text files and treated as string in python 2 and python 3 avoiding the error.

run_command with universal_newlines=True:

When using python 3 output type: <class 'str'> and no error.

When using python 2 output type: <type 'str'> and no error.


run_command as it is :
When using python 3 output type: <class 'bytes'> and TypeError.

When using python 2 output type: <type 'str'> and no error.


  was:
The problem is: if you use python3 and the '_cmd' on the config file airflow will not start due the error:
File "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py", line 447, in _validate
    "sqlite" in self.get('core', 'sql_alchemy_conn')):
TypeError: 'str' does not support the buffer interface

To reproduce the problem change the following line on airflow.cfg:
sql_alchemy_conn_cmd = echo sqlite:////~/airflow/airflow.db

The solution is change the following run_command method's line on configuration.py:
command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)

By setting the universal_newlines to true the file objects stdout and stderr are opened as text files and treated as string in python 2 and python 3 avoiding the error.

run_command with universal_newlines=True:

When using python 3 output type: <class 'str'> and no error.

When using python 2 output type: <type 'str'> and no error.


run_command as it is :
When using python 3 output type: <class 'bytes'> and TypeError.

When using python 2 output type: <type 'str'> and no error.


> Configuration parsing doesn't work properly with python 3
> ---------------------------------------------------------
>
>                 Key: AIRFLOW-668
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-668
>             Project: Apache Airflow
>          Issue Type: Bug
>         Environment:  Airflow version: v1.7.1.3
> - Airflow components: webserver and scheduler with a postgres database and CeleryExecutor
> - Python Version: 3.4.5
>            Reporter: Rafael Gomes Fernandes
>            Assignee: Rafael Gomes Fernandes
>
> The problem is: if you use python3 and the '_cmd' on the config file airflow will not start due the error:
> {noformat}
> File "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py", line 447, in _validate
>     "sqlite" in self.get('core', 'sql_alchemy_conn')):
> TypeError: 'str' does not support the buffer interface
> {noformat}
> To reproduce the problem change the following line on airflow.cfg:
> {code:title=airflow.cfgborderStyle=solid}
> sql_alchemy_conn_cmd = echo sqlite:////~/airflow/airflow.db
> {code}
> The solution is change the following run_command method's line on configuration.py:
> {code:title=airflow/configuration.py|borderStyle=solid}
> def run_command(command):
>     """
>     Runs command and returns stdout
>     """
>     process = subprocess.Popen(
>         command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE{color:green}, universal_newlines=True{color})
>     output, stderr = process.communicate()
>     if process.returncode != 0:
>         raise AirflowConfigException(
>             "Cannot execute {}. Error code is: {}. Output: {}, Stderr: {}"
>             .format(command, process.returncode, output, stderr)
>         )
>     return output
> {code}
> By setting the universal_newlines to true the file objects stdout and stderr are opened as text files and treated as string in python 2 and python 3 avoiding the error.
> run_command with universal_newlines=True:
> When using python 3 output type: <class 'str'> and no error.
> When using python 2 output type: <type 'str'> and no error.
> run_command as it is :
> When using python 3 output type: <class 'bytes'> and TypeError.
> When using python 2 output type: <type 'str'> and no error.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)