Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
Date: Wed, 18 May 2016 17:14:13 +0000 (UTC)
From: "Raul Landa (JIRA)" <jira@apache.org>
To: commits@airflow.incubator.apache.org
Message-ID: <JIRA.12970841.1463591557000.225840.1463591653044@Atlassian.JIRA>
In-Reply-To: <JIRA.12970841.1463591557000@Atlassian.JIRA>
References: <JIRA.12970841.1463591557000@Atlassian.JIRA> <JIRA.12970841.1463591557886@arcas>
Subject: [jira] [Updated] (AIRFLOW-130) SSHHook fails in OSX due to mktemp
 implementation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 18 May 2016 17:14:18 -0000


     [ https://issues.apache.org/jira/browse/AIRFLOW-130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Raul Landa updated AIRFLOW-130:
-------------------------------
    Environment: 
Airflow components: 
webserver and scheduler with a mysql database and LocalExecutor

Relevant `airflow.cfg` settings:
ssh_default connection correctly configured

Python Version: 
Python 2.7.11

Operating System: 
Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64 x86_64

Python packages: 
airflow (1.7.0)
alembic (0.8.6)
Babel (1.3)
cffi (1.6.0)
chartkick (0.4.2)
croniter (0.3.12)
cryptography (1.3.2)
dill (0.2.5)
enum34 (1.1.6)
Flask (0.10.1)
Flask-Admin (1.4.0)
Flask-Cache (0.13.1)
Flask-Login (0.2.11)
Flask-WTF (0.12)
future (0.15.2)
gunicorn (19.3.0)
idna (2.1)
ipaddress (1.0.16)
itsdangerous (0.24)
Jinja2 (2.8)
Mako (1.0.4)
Markdown (2.6.6)
MarkupSafe (0.23)
mysqlclient (1.3.7)
netaddr (0.7.18)
numpy (1.11.0)
pandas (0.18.1)
pip (8.1.2)
pyasn1 (0.1.9)
pycparser (2.14)
Pygments (2.1.3)
python-dateutil (2.5.3)
python-editor (1.0)
pytz (2016.4)
requests (2.10.0)
setproctitle (1.1.10)
setuptools (21.0.0)
six (1.10.0)
slackclient (1.0.0)
SQLAlchemy (1.0.13)
thrift (0.9.3)
websocket-client (0.37.0)
Werkzeug (0.11.9)
wheel (0.29.0)
WTForms (2.1)


  was:
* Airflow components: webserver and scheduler with a mysql database and LocalExecutor
* Relevant `airflow.cfg` settings:
ssh_default connection correctly configured
* Python Version: Python 2.7.11
* Operating System: 
Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64 x86_64

* Python packages: 
airflow (1.7.0)
alembic (0.8.6)
Babel (1.3)
cffi (1.6.0)
chartkick (0.4.2)
croniter (0.3.12)
cryptography (1.3.2)
dill (0.2.5)
enum34 (1.1.6)
Flask (0.10.1)
Flask-Admin (1.4.0)
Flask-Cache (0.13.1)
Flask-Login (0.2.11)
Flask-WTF (0.12)
future (0.15.2)
gunicorn (19.3.0)
idna (2.1)
ipaddress (1.0.16)
itsdangerous (0.24)
Jinja2 (2.8)
Mako (1.0.4)
Markdown (2.6.6)
MarkupSafe (0.23)
mysqlclient (1.3.7)
netaddr (0.7.18)
numpy (1.11.0)
pandas (0.18.1)
pip (8.1.2)
pyasn1 (0.1.9)
pycparser (2.14)
Pygments (2.1.3)
python-dateutil (2.5.3)
python-editor (1.0)
pytz (2016.4)
requests (2.10.0)
setproctitle (1.1.10)
setuptools (21.0.0)
six (1.10.0)
slackclient (1.0.0)
SQLAlchemy (1.0.13)
thrift (0.9.3)
websocket-client (0.37.0)
Werkzeug (0.11.9)
wheel (0.29.0)
WTForms (2.1)


> SSHHook fails in OSX due to mktemp implementation
> -------------------------------------------------
>
>                 Key: AIRFLOW-130
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-130
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: hooks
>    Affects Versions: Airflow 1.7.0
>         Environment: Airflow components: 
> webserver and scheduler with a mysql database and LocalExecutor
> Relevant `airflow.cfg` settings:
> ssh_default connection correctly configured
> Python Version: 
> Python 2.7.11
> Operating System: 
> Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64 x86_64
> Python packages: 
> airflow (1.7.0)
> alembic (0.8.6)
> Babel (1.3)
> cffi (1.6.0)
> chartkick (0.4.2)
> croniter (0.3.12)
> cryptography (1.3.2)
> dill (0.2.5)
> enum34 (1.1.6)
> Flask (0.10.1)
> Flask-Admin (1.4.0)
> Flask-Cache (0.13.1)
> Flask-Login (0.2.11)
> Flask-WTF (0.12)
> future (0.15.2)
> gunicorn (19.3.0)
> idna (2.1)
> ipaddress (1.0.16)
> itsdangerous (0.24)
> Jinja2 (2.8)
> Mako (1.0.4)
> Markdown (2.6.6)
> MarkupSafe (0.23)
> mysqlclient (1.3.7)
> netaddr (0.7.18)
> numpy (1.11.0)
> pandas (0.18.1)
> pip (8.1.2)
> pyasn1 (0.1.9)
> pycparser (2.14)
> Pygments (2.1.3)
> python-dateutil (2.5.3)
> python-editor (1.0)
> pytz (2016.4)
> requests (2.10.0)
> setproctitle (1.1.10)
> setuptools (21.0.0)
> six (1.10.0)
> slackclient (1.0.0)
> SQLAlchemy (1.0.13)
> thrift (0.9.3)
> websocket-client (0.37.0)
> Werkzeug (0.11.9)
> wheel (0.29.0)
> WTForms (2.1)
>            Reporter: Raul Landa
>
> h3. Description of Issue
> * What did you expect to happen?
> In the {{__enter__}} method in the {{SSHTempFileContent}} class ({{ssh_execute_operator.py}}) a file should be created using {{mktemp}}
> * What happened instead?
> The call to mktemp fails, because the {{--tmpdir}} command line option is not supported in OSX
> (see https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/mktemp.1.html)
> * Stack trace, if appropriate:
> {noformat}
> [2016-05-18 17:39:31,735] {models.py:1144} ERROR - Failed to create remote temp file
> Traceback (most recent call last):
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/bin/airflow", line 15, in <module>
>     args.func(args)
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/bin/cli.py", line 316, in test
>     ti.run(force=True, ignore_dependencies=True, test_mode=True)
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/models.py", line 1067, in run
>     result = task_copy.execute(context=context)
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py", line 116, in execute
>     self.task_id) as remote_file_path:
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py", line 52, in __enter__
>     raise AirflowException("Failed to create remote temp file")
> airflow.utils.AirflowException: Failed to create remote temp file
> {noformat}
> The stack trace is not very useful. However looking at the tuple returned by {{pmktemp.communicate()}} is helpful:
> {noformat}
> mktemp: illegal option -- -
> usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
>        mktemp [-d] [-q] [-u] -t prefix 
> {noformat}
> h3. Reproducing the Issue
> Issue happens with normal usage of {{SSHHook}} when attempting to run commands on an OSX box.
> h3. Fix
> Changing {{--tmpdir}} by {{-t}} seems to fix the issue in OSX, but I do not have access to a linux box to check if it breaks there.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)