Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
Date: Wed, 18 May 2016 22:13:12 +0000 (UTC)
From: "Chris Riccomini (JIRA)" <jira@apache.org>
To: commits@airflow.incubator.apache.org
Message-ID: <JIRA.12970841.1463591557000.229650.1463609592861@Atlassian.JIRA>
In-Reply-To: <JIRA.12970841.1463591557000@Atlassian.JIRA>
References: <JIRA.12970841.1463591557000@Atlassian.JIRA> <JIRA.12970841.1463591557886@arcas>
Subject: [jira] [Commented] (AIRFLOW-130) SSHHook fails in OSX due to mktemp
 implementation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 18 May 2016 22:13:18 -0000


    [ https://issues.apache.org/jira/browse/AIRFLOW-130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289960#comment-15289960 ] 

Chris Riccomini commented on AIRFLOW-130:
-----------------------------------------

Using -t appears to work. I've confirmed:

CentOS:

{noformat}
$ mktemp -t bar_XXXXXX
/tmp/bar_LaJ3Wp
{noformat}

Mac:

{noformat}
$ mktemp -t bar_XXXXXX
/var/folders/tc/1fml66yd3jjg_mj6z601kbcs9h5tkc/T/bar_XXXXXX.RGCIn3dS
{noformat}

One oddity is that the XXXXXX is not required (or works differently) on Mac, but no big deal.

[~rlanda], want to send a PR?

> SSHHook fails in OSX due to mktemp implementation
> -------------------------------------------------
>
>                 Key: AIRFLOW-130
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-130
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: hooks
>    Affects Versions: Airflow 1.7.0
>         Environment: Airflow components: 
> webserver and scheduler with a mysql database and LocalExecutor
> Relevant `airflow.cfg` settings:
> ssh_default connection correctly configured
> Python Version: 
> Python 2.7.11
> Operating System: 
> Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64 x86_64
> Python packages: 
> airflow (1.7.0)
> alembic (0.8.6)
> Babel (1.3)
> cffi (1.6.0)
> chartkick (0.4.2)
> croniter (0.3.12)
> cryptography (1.3.2)
> dill (0.2.5)
> enum34 (1.1.6)
> Flask (0.10.1)
> Flask-Admin (1.4.0)
> Flask-Cache (0.13.1)
> Flask-Login (0.2.11)
> Flask-WTF (0.12)
> future (0.15.2)
> gunicorn (19.3.0)
> idna (2.1)
> ipaddress (1.0.16)
> itsdangerous (0.24)
> Jinja2 (2.8)
> Mako (1.0.4)
> Markdown (2.6.6)
> MarkupSafe (0.23)
> mysqlclient (1.3.7)
> netaddr (0.7.18)
> numpy (1.11.0)
> pandas (0.18.1)
> pip (8.1.2)
> pyasn1 (0.1.9)
> pycparser (2.14)
> Pygments (2.1.3)
> python-dateutil (2.5.3)
> python-editor (1.0)
> pytz (2016.4)
> requests (2.10.0)
> setproctitle (1.1.10)
> setuptools (21.0.0)
> six (1.10.0)
> slackclient (1.0.0)
> SQLAlchemy (1.0.13)
> thrift (0.9.3)
> websocket-client (0.37.0)
> Werkzeug (0.11.9)
> wheel (0.29.0)
> WTForms (2.1)
>            Reporter: Raul Landa
>
> h3. Description of Issue
> * What did you expect to happen?
> In the {{__enter__}} method in the {{SSHTempFileContent}} class ({{ssh_execute_operator.py}}) a file should be created using {{mktemp}}
> * What happened instead?
> The call to mktemp fails, because the {{--tmpdir}} command line option is not supported in OSX
> (see https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/mktemp.1.html)
> * Stack trace, if appropriate:
> {noformat}
> [2016-05-18 17:39:31,735] {models.py:1144} ERROR - Failed to create remote temp file
> Traceback (most recent call last):
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/bin/airflow", line 15, in <module>
>     args.func(args)
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/bin/cli.py", line 316, in test
>     ti.run(force=True, ignore_dependencies=True, test_mode=True)
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/models.py", line 1067, in run
>     result = task_copy.execute(context=context)
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py", line 116, in execute
>     self.task_id) as remote_file_path:
>   File "/Users/rlanda/.virtualenvs/datapath-airflow/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py", line 52, in __enter__
>     raise AirflowException("Failed to create remote temp file")
> airflow.utils.AirflowException: Failed to create remote temp file
> {noformat}
> The stack trace is not very useful. However looking at the tuple returned by {{pmktemp.communicate()}} is helpful:
> {noformat}
> mktemp: illegal option -- -
> usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
>        mktemp [-d] [-q] [-u] -t prefix 
> {noformat}
> h3. Reproducing the Issue
> Issue happens with normal usage of {{SSHHook}} when attempting to run commands on an OSX box.
> h3. Fix
> Changing {{--tmpdir}} by {{-t}} seems to fix the issue in OSX, but I do not have access to a linux box to check if it breaks there.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)