From commits-return-14551-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Thu May 3 22:55:36 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id BBBB6180625 for ; Thu, 3 May 2018 22:55:35 +0200 (CEST) Received: (qmail 22922 invoked by uid 500); 3 May 2018 20:55:34 -0000 Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list commits@airflow.incubator.apache.org Received: (qmail 22909 invoked by uid 99); 3 May 2018 20:55:34 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 May 2018 20:55:34 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 5896AC8A33 for ; Thu, 3 May 2018 20:55:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -11.701 X-Spam-Level: X-Spam-Status: No, score=-11.701 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_HI=-5, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id y_Vmj3nQSKQL for ; Thu, 3 May 2018 20:55:33 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id 303865FBEC for ; Thu, 3 May 2018 20:55:32 +0000 (UTC) Received: (qmail 22886 invoked by uid 99); 3 May 2018 20:55:31 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 May 2018 20:55:31 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 45E3EE184D; Thu, 3 May 2018 20:55:31 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: fokko@apache.org To: commits@airflow.incubator.apache.org Message-Id: <5e64b38f17c14207aa42c1d5868c9413@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: incubator-airflow git commit: [AIRFLOW-2411] add dataproc_jars to templated_fields Date: Thu, 3 May 2018 20:55:31 +0000 (UTC) Repository: incubator-airflow Updated Branches: refs/heads/master 6b3f6cee1 -> c5fa8cd41 [AIRFLOW-2411] add dataproc_jars to templated_fields This commit makes it possible to use jinja templates when passing JAR file URIs to the DataProc operators that require JAR files, specifically the DataProc Hive, Pig, SparkSql, Spark, Hadoop and PySpark operators. Closes #3305 from mchalek/template-dataproc-jars Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c5fa8cd4 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c5fa8cd4 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c5fa8cd4 Branch: refs/heads/master Commit: c5fa8cd411ca67889eafd109988d2472ccfbba10 Parents: 6b3f6ce Author: Kevin McHale Authored: Thu May 3 22:55:10 2018 +0200 Committer: Fokko Driesprong Committed: Thu May 3 22:55:10 2018 +0200 ---------------------------------------------------------------------- airflow/contrib/operators/dataproc_operator.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c5fa8cd4/airflow/contrib/operators/dataproc_operator.py ---------------------------------------------------------------------- diff --git a/airflow/contrib/operators/dataproc_operator.py b/airflow/contrib/operators/dataproc_operator.py index 56ebb91..ad0aa09 100644 --- a/airflow/contrib/operators/dataproc_operator.py +++ b/airflow/contrib/operators/dataproc_operator.py @@ -481,7 +481,7 @@ class DataProcPigOperator(BaseOperator): :param region: The specified region where the dataproc cluster is created. :type region: string """ - template_fields = ['query', 'variables', 'job_name', 'cluster_name'] + template_fields = ['query', 'variables', 'job_name', 'cluster_name', 'dataproc_jars'] template_ext = ('.pg', '.pig',) ui_color = '#0273d4' @@ -561,7 +561,7 @@ class DataProcHiveOperator(BaseOperator): :param region: The specified region where the dataproc cluster is created. :type region: string """ - template_fields = ['query', 'variables', 'job_name', 'cluster_name'] + template_fields = ['query', 'variables', 'job_name', 'cluster_name', 'dataproc_jars'] template_ext = ('.q',) ui_color = '#0273d4' @@ -642,7 +642,7 @@ class DataProcSparkSqlOperator(BaseOperator): :param region: The specified region where the dataproc cluster is created. :type region: string """ - template_fields = ['query', 'variables', 'job_name', 'cluster_name'] + template_fields = ['query', 'variables', 'job_name', 'cluster_name', 'dataproc_jars'] template_ext = ('.q',) ui_color = '#0273d4' @@ -731,7 +731,7 @@ class DataProcSparkOperator(BaseOperator): :type region: string """ - template_fields = ['arguments', 'job_name', 'cluster_name'] + template_fields = ['arguments', 'job_name', 'cluster_name', 'dataproc_jars'] ui_color = '#0273d4' @apply_defaults @@ -821,7 +821,7 @@ class DataProcHadoopOperator(BaseOperator): :type region: string """ - template_fields = ['arguments', 'job_name', 'cluster_name'] + template_fields = ['arguments', 'job_name', 'cluster_name', 'dataproc_jars'] ui_color = '#0273d4' @apply_defaults @@ -911,7 +911,7 @@ class DataProcPySparkOperator(BaseOperator): :type region: string """ - template_fields = ['arguments', 'job_name', 'cluster_name'] + template_fields = ['arguments', 'job_name', 'cluster_name', 'dataproc_jars'] ui_color = '#0273d4' @staticmethod