airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-5118) Airflow DataprocClusterCreateOperator does not currently support setting optional components
Date Wed, 14 Aug 2019 18:35:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-5118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907528#comment-16907528
] 

ASF GitHub Bot commented on AIRFLOW-5118:
-----------------------------------------

idralyuk commented on pull request #5820: [AIRFLOW-5118] Add ability to specify optional components
in Dataproc…
URL: https://github.com/apache/airflow/pull/5820
 
 
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Airflow DataprocClusterCreateOperator does not currently support setting optional components
> --------------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-5118
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5118
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: operators
>    Affects Versions: 1.10.3
>            Reporter: Omid Vahdaty
>            Assignee: Igor
>            Priority: Minor
>
> there need to be an option to install optional components via DataprocClusterCreateOperator
. components such as zeppelin.
> From the source code of the DataprocClusterCreateOperator[1], the only software configs
that can be set are the imageVersion and the properties. As the Zeppelin component needs to
be set through softwareConfig optionalComponents[2], the DataprocClusterCreateOperator does
not currently support setting optional components. 
>  
> As a workaround for the time being, you could create your clusters by directly using
the gcloud command rather than the DataprocClusterCreateOperator . Using the Airflow BashOperator[4],
you can execute gcloud commands that create your Dataproc cluster with the required optional
components. 
> [1] [https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py] 
>  [2] [https://cloud.google.com/dataproc/docs/reference/rest/v1/ClusterConfig#softwareconfig] 
> [3] [https://airflow.apache.org/howto/operator/bash.html] 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Mime
View raw message