airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raul Landa (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-130) SSHHook fails in OSX due to mktemp implementation
Date Wed, 18 May 2016 17:13:12 GMT
Raul Landa created AIRFLOW-130:
----------------------------------

             Summary: SSHHook fails in OSX due to mktemp implementation
                 Key: AIRFLOW-130
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-130
             Project: Apache Airflow
          Issue Type: Bug
          Components: hooks
    Affects Versions: Airflow 1.7.0
         Environment: * Airflow components: webserver and scheduler with a mysql database
and LocalExecutor
* Relevant `airflow.cfg` settings:
ssh_default connection correctly configured
* Python Version: Python 2.7.11
* Operating System: 
Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64
x86_64

* Python packages: 
airflow (1.7.0)
alembic (0.8.6)
Babel (1.3)
cffi (1.6.0)
chartkick (0.4.2)
croniter (0.3.12)
cryptography (1.3.2)
dill (0.2.5)
enum34 (1.1.6)
Flask (0.10.1)
Flask-Admin (1.4.0)
Flask-Cache (0.13.1)
Flask-Login (0.2.11)
Flask-WTF (0.12)
future (0.15.2)
gunicorn (19.3.0)
idna (2.1)
ipaddress (1.0.16)
itsdangerous (0.24)
Jinja2 (2.8)
Mako (1.0.4)
Markdown (2.6.6)
MarkupSafe (0.23)
mysqlclient (1.3.7)
netaddr (0.7.18)
numpy (1.11.0)
pandas (0.18.1)
pip (8.1.2)
pyasn1 (0.1.9)
pycparser (2.14)
Pygments (2.1.3)
python-dateutil (2.5.3)
python-editor (1.0)
pytz (2016.4)
requests (2.10.0)
setproctitle (1.1.10)
setuptools (21.0.0)
six (1.10.0)
slackclient (1.0.0)
SQLAlchemy (1.0.13)
thrift (0.9.3)
websocket-client (0.37.0)
Werkzeug (0.11.9)
wheel (0.29.0)
WTForms (2.1)

            Reporter: Raul Landa


h3. Description of Issue

* What did you expect to happen?

In the {{__enter__}} method in the {{SSHTempFileContent}} class ({{ssh_execute_operator.py}})
a file should be created using {{mktemp}}

* What happened instead?

The call to mktemp fails, because the {{--tmpdir}} command line option is not supported in
OSX
(see https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/mktemp.1.html)

* Stack trace, if appropriate:

{noformat}
[2016-05-18 17:39:31,735] {models.py:1144} ERROR - Failed to create remote temp file
Traceback (most recent call last):
  File "/Users/rlanda/.virtualenvs/datapath-airflow/bin/airflow", line 15, in <module>
    args.func(args)
  File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/bin/cli.py",
line 316, in test
    ti.run(force=True, ignore_dependencies=True, test_mode=True)
  File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/models.py",
line 1067, in run
    result = task_copy.execute(context=context)
  File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py",
line 116, in execute
    self.task_id) as remote_file_path:
  File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py",
line 52, in __enter__
    raise AirflowException("Failed to create remote temp file")
airflow.utils.AirflowException: Failed to create remote temp file
{noformat}

The stack trace is not very useful. However looking at the tuple returned by {{pmktemp.communicate()}}
is helpful:

{noformat}
mktemp: illegal option -- -
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix 
{noformat}



h3. Reproducing the Issue

Issue happens with normal usage of {{SSHHook}} when attempting to run commands on an OSX box.


h3. Fix

Changing {{--tmpdir}} by {{-t}} seems to fix the issue in OSX, but I do not have access to
a linux box to check if it breaks there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message