airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-705) airflow.configuration.run_command output does not decode command output
Date Thu, 16 Mar 2017 23:43:42 GMT


ASF subversion and git services commented on AIRFLOW-705:

Commit 0bb6f2f458ddf305d77fc63d5e817c9b6eb22270 in incubator-airflow's branch refs/heads/master
from [~thesquelched]
[;h=0bb6f2f ]

[AIRFLOW-705][AIRFLOW-706] Fix run_command bugs

`run_command` currently uses `str.split` for shell
commands, but this
breaks commands like `echo "foo bar"`.  Instead,
use `shlex.split`,
which understands how to handle quotes.

In python3, subprocess functions return
stdout/stderr as `bytes`, not
`str`.  To fix py2/py3 compatibility,
`run_command` now decodes the
output so that it always returns a `str`.

Dear Airflow Maintainers,

Please accept this PR that addresses the following

Testing Done:
- Changed `sql_alchemy_conn` to
`sql_alchemy_conn_cmd` in test configuration to
exercise `run_command` in tests

Closes #2053 from thesquelched/AIRFLOW-706

> airflow.configuration.run_command output does not decode command output
> -----------------------------------------------------------------------
>                 Key: AIRFLOW-705
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Bug
>    Affects Versions: Airflow
>            Reporter: Scott Kruger
> In python3, command outputs (to {{stdout}} and {{stderr}}) are of type {{bytes}}, not
{{str}}.  This causes the following exception when the {{sql_alchemy_conn_cmd}} option is
> {noformat}
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 4, in <module>
>     from airflow import configuration
>   File "/usr/local/lib/python3.5/site-packages/airflow/", line 29, in <module>
>     from airflow import configuration as conf
>   File "/usr/local/lib/python3.5/site-packages/airflow/", line 657, in
>   File "/usr/local/lib/python3.5/site-packages/airflow/", line 519, in
>     self._validate()
>   File "/usr/local/lib/python3.5/site-packages/airflow/", line 447, in
>     "sqlite" in self.get('core', 'sql_alchemy_conn')):
> TypeError: a bytes-like object is required, not 'str'
> {noformat}
> For compatibility between python 2 and 3, {{airflow.configuration.run_command}} should
decode {{stdout}} to produce a (unicode) string.

This message was sent by Atlassian JIRA

View raw message