airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luis Angel Vicente Sanchez (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-1409) SparkSubmitHook doesn't generate a valid master URL
Date Thu, 13 Jul 2017 11:44:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Luis Angel Vicente Sanchez updated AIRFLOW-1409:
------------------------------------------------
    Description: 
If you configure the connection using an ENV variable (and possibly through the UI), SparkSubmitHook
is ignoring the scheme and doesn't produce valid master URL. Example, this ENV variable,

AIRFLOW_CONN_SPARK_CLUSTER=spark://spark-master:7077 

will produce the following master URL:

spark-master:7077

This URL is not valid and the job submission fails. The issue is how the url is parsed to
create a connection, I have a partial fix working and I will submit a PR to github soon.

  was:
If you configure the connection using an ENV variable (and possibly through the UI), SparkSubmitHook
is ignoring the scheme and doesn't produce valid master URL. Example, this ENV variable,

AIRFLOW_CONN_SPARK_CLUSTER=spark://spark-master:7077 

will produce the following master URL:

spark-master:7077

This URL is not valid and the job submission fails (the same happens with a mesos master).

I will create a PR on github that fixes this.


> SparkSubmitHook doesn't generate a valid master URL
> ---------------------------------------------------
>
>                 Key: AIRFLOW-1409
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1409
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Luis Angel Vicente Sanchez
>
> If you configure the connection using an ENV variable (and possibly through the UI),
SparkSubmitHook is ignoring the scheme and doesn't produce valid master URL. Example, this
ENV variable,
> AIRFLOW_CONN_SPARK_CLUSTER=spark://spark-master:7077 
> will produce the following master URL:
> spark-master:7077
> This URL is not valid and the job submission fails. The issue is how the url is parsed
to create a connection, I have a partial fix working and I will submit a PR to github soon.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message