airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ash Berlin-Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-1326) CLONE - Contrib Spark Submit Hook _should not_ split argument array
Date Tue, 20 Jun 2017 10:32:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-1326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ash Berlin-Taylor updated AIRFLOW-1326:
---------------------------------------
    Description: 
AIRFLOW-1184 introduced a bug making it impossible to include an argument with an embedded
space in the SparkSubmitOperator. (I believe the original bug report should have been closed
as wontfix/not-a-bug as it was user error in how the SparkSubmitOperator is being called)

i.e. the correct way to achieve what was wanted in the bug is like this:

{code}
            'application_args': [
                '-f', ' foo',
                '--bar', 'bar',
                '--start', '{{ macros.ds_add(ds, -1)}}',
                '--end', '{{ ds }}'
            ]
{code}

This was possibly compounded as the tests were showing something that would never "work"

  was:
Python Popen expect a list as command. Spark submit too, as: 
* ['--option value'] 
is not the same as 
* ['--option', 'value']

in regards of spark. eg spark logs : (yarn logs)

Error: Unknown option --end 2017-05-08
Error: Unknown option --begin 2017-05-07
Error: Unknown option --db_name mydb
Error: Missing option --begin
Error: Missing option --end
Error: Missing option --db_name



> CLONE - Contrib Spark Submit Hook _should not_ split argument array
> -------------------------------------------------------------------
>
>                 Key: AIRFLOW-1326
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1326
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: contrib, hooks
>    Affects Versions: Airflow 2.0, Airflow 1.8
>            Reporter: Ash Berlin-Taylor
>            Assignee: Vianney FOUCAULT
>             Fix For: Airflow 2.0, Airflow 1.8
>
>
> AIRFLOW-1184 introduced a bug making it impossible to include an argument with an embedded
space in the SparkSubmitOperator. (I believe the original bug report should have been closed
as wontfix/not-a-bug as it was user error in how the SparkSubmitOperator is being called)
> i.e. the correct way to achieve what was wanted in the bug is like this:
> {code}
>             'application_args': [
>                 '-f', ' foo',
>                 '--bar', 'bar',
>                 '--start', '{{ macros.ds_add(ds, -1)}}',
>                 '--end', '{{ ds }}'
>             ]
> {code}
> This was possibly compounded as the tests were showing something that would never "work"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message