airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luca Falsina (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-1350) Add "query_uri" parameter for Google DataProc operator for both SparkSQL and Hive jobs
Date Tue, 27 Jun 2017 14:01:00 GMT
Luca Falsina created AIRFLOW-1350:
-------------------------------------

             Summary: Add "query_uri" parameter for Google DataProc operator for both SparkSQL
and Hive jobs
                 Key: AIRFLOW-1350
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1350
             Project: Apache Airflow
          Issue Type: Improvement
          Components: gcp, operators
            Reporter: Luca Falsina
            Assignee: Luca Falsina


Google DataProc APIs allow to provide as an input for some types of job (Pig, SparkSQL, Hive)
a "query_file" parameter pointing to a file stored in Google Cloud Storage.

The current implementation of the DataProc operator allows this functionality only for Pig
jobs. The code modification should be trivial as I plan to just reuse the same implementation
in SparkSQL and Hive operator.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message