airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raul Landa (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-130) SSHHook fails in OSX due to mktemp implementation
Date Wed, 18 May 2016 17:14:13 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Raul Landa updated AIRFLOW-130:
-------------------------------
    Environment: 
Airflow components: 
webserver and scheduler with a mysql database and LocalExecutor

Relevant `airflow.cfg` settings:
ssh_default connection correctly configured

Python Version: 
Python 2.7.11

Operating System: 
Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64
x86_64

Python packages: 
airflow (1.7.0)
alembic (0.8.6)
Babel (1.3)
cffi (1.6.0)
chartkick (0.4.2)
croniter (0.3.12)
cryptography (1.3.2)
dill (0.2.5)
enum34 (1.1.6)
Flask (0.10.1)
Flask-Admin (1.4.0)
Flask-Cache (0.13.1)
Flask-Login (0.2.11)
Flask-WTF (0.12)
future (0.15.2)
gunicorn (19.3.0)
idna (2.1)
ipaddress (1.0.16)
itsdangerous (0.24)
Jinja2 (2.8)
Mako (1.0.4)
Markdown (2.6.6)
MarkupSafe (0.23)
mysqlclient (1.3.7)
netaddr (0.7.18)
numpy (1.11.0)
pandas (0.18.1)
pip (8.1.2)
pyasn1 (0.1.9)
pycparser (2.14)
Pygments (2.1.3)
python-dateutil (2.5.3)
python-editor (1.0)
pytz (2016.4)
requests (2.10.0)
setproctitle (1.1.10)
setuptools (21.0.0)
six (1.10.0)
slackclient (1.0.0)
SQLAlchemy (1.0.13)
thrift (0.9.3)
websocket-client (0.37.0)
Werkzeug (0.11.9)
wheel (0.29.0)
WTForms (2.1)


  was:
* Airflow components: webserver and scheduler with a mysql database and LocalExecutor
* Relevant `airflow.cfg` settings:
ssh_default connection correctly configured
* Python Version: Python 2.7.11
* Operating System: 
Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64
x86_64

* Python packages: 
airflow (1.7.0)
alembic (0.8.6)
Babel (1.3)
cffi (1.6.0)
chartkick (0.4.2)
croniter (0.3.12)
cryptography (1.3.2)
dill (0.2.5)
enum34 (1.1.6)
Flask (0.10.1)
Flask-Admin (1.4.0)
Flask-Cache (0.13.1)
Flask-Login (0.2.11)
Flask-WTF (0.12)
future (0.15.2)
gunicorn (19.3.0)
idna (2.1)
ipaddress (1.0.16)
itsdangerous (0.24)
Jinja2 (2.8)
Mako (1.0.4)
Markdown (2.6.6)
MarkupSafe (0.23)
mysqlclient (1.3.7)
netaddr (0.7.18)
numpy (1.11.0)
pandas (0.18.1)
pip (8.1.2)
pyasn1 (0.1.9)
pycparser (2.14)
Pygments (2.1.3)
python-dateutil (2.5.3)
python-editor (1.0)
pytz (2016.4)
requests (2.10.0)
setproctitle (1.1.10)
setuptools (21.0.0)
six (1.10.0)
slackclient (1.0.0)
SQLAlchemy (1.0.13)
thrift (0.9.3)
websocket-client (0.37.0)
Werkzeug (0.11.9)
wheel (0.29.0)
WTForms (2.1)



> SSHHook fails in OSX due to mktemp implementation
> -------------------------------------------------
>
>                 Key: AIRFLOW-130
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-130
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: hooks
>    Affects Versions: Airflow 1.7.0
>         Environment: Airflow components: 
> webserver and scheduler with a mysql database and LocalExecutor
> Relevant `airflow.cfg` settings:
> ssh_default connection correctly configured
> Python Version: 
> Python 2.7.11
> Operating System: 
> Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64
x86_64
> Python packages: 
> airflow (1.7.0)
> alembic (0.8.6)
> Babel (1.3)
> cffi (1.6.0)
> chartkick (0.4.2)
> croniter (0.3.12)
> cryptography (1.3.2)
> dill (0.2.5)
> enum34 (1.1.6)
> Flask (0.10.1)
> Flask-Admin (1.4.0)
> Flask-Cache (0.13.1)
> Flask-Login (0.2.11)
> Flask-WTF (0.12)
> future (0.15.2)
> gunicorn (19.3.0)
> idna (2.1)
> ipaddress (1.0.16)
> itsdangerous (0.24)
> Jinja2 (2.8)
> Mako (1.0.4)
> Markdown (2.6.6)
> MarkupSafe (0.23)
> mysqlclient (1.3.7)
> netaddr (0.7.18)
> numpy (1.11.0)
> pandas (0.18.1)
> pip (8.1.2)
> pyasn1 (0.1.9)
> pycparser (2.14)
> Pygments (2.1.3)
> python-dateutil (2.5.3)
> python-editor (1.0)
> pytz (2016.4)
> requests (2.10.0)
> setproctitle (1.1.10)
> setuptools (21.0.0)
> six (1.10.0)
> slackclient (1.0.0)
> SQLAlchemy (1.0.13)
> thrift (0.9.3)
> websocket-client (0.37.0)
> Werkzeug (0.11.9)
> wheel (0.29.0)
> WTForms (2.1)
>            Reporter: Raul Landa
>
> h3. Description of Issue
> * What did you expect to happen?
> In the {{__enter__}} method in the {{SSHTempFileContent}} class ({{ssh_execute_operator.py}})
a file should be created using {{mktemp}}
> * What happened instead?
> The call to mktemp fails, because the {{--tmpdir}} command line option is not supported
in OSX
> (see https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/mktemp.1.html)
> * Stack trace, if appropriate:
> {noformat}
> [2016-05-18 17:39:31,735] {models.py:1144} ERROR - Failed to create remote temp file
> Traceback (most recent call last):
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/bin/airflow", line 15, in <module>
>     args.func(args)
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/bin/cli.py",
line 316, in test
>     ti.run(force=True, ignore_dependencies=True, test_mode=True)
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/models.py",
line 1067, in run
>     result = task_copy.execute(context=context)
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py",
line 116, in execute
>     self.task_id) as remote_file_path:
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py",
line 52, in __enter__
>     raise AirflowException("Failed to create remote temp file")
> airflow.utils.AirflowException: Failed to create remote temp file
> {noformat}
> The stack trace is not very useful. However looking at the tuple returned by {{pmktemp.communicate()}}
is helpful:
> {noformat}
> mktemp: illegal option -- -
> usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
>        mktemp [-d] [-q] [-u] -t prefix 
> {noformat}
> h3. Reproducing the Issue
> Issue happens with normal usage of {{SSHHook}} when attempting to run commands on an
OSX box.
> h3. Fix
> Changing {{--tmpdir}} by {{-t}} seems to fix the issue in OSX, but I do not have access
to a linux box to check if it breaks there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message