airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] andrewmchen commented on a change in pull request #3813: [AIRFLOW-1998] Implemented DatabricksRunNowOperator for jobs/run-now …
Date Tue, 28 Aug 2018 21:12:51 GMT
andrewmchen commented on a change in pull request #3813: [AIRFLOW-1998] Implemented DatabricksRunNowOperator
for jobs/run-now …
URL: https://github.com/apache/incubator-airflow/pull/3813#discussion_r213470207
 
 

 ##########
 File path: airflow/contrib/operators/databricks_operator.py
 ##########
 @@ -30,6 +30,81 @@
 XCOM_RUN_PAGE_URL_KEY = 'run_page_url'
 
 
+def _deep_string_coerce(content, json_path='json'):
+    """
+    Coerces content or all values of content if it is a dict to a string. The
+    function will throw if content contains non-string or non-numeric types.
+
+    The reason why we have this function is because the ``self.json`` field must be a
+     dict with only string values. This is because ``render_template`` will fail
+    for numerical values.
+    """
+    c = _deep_string_coerce
+    if isinstance(content, six.string_types):
+        return content
+    elif isinstance(content, six.integer_types + (float,)):
+        # Databricks can tolerate either numeric or string types in the API backend.
+        return str(content)
+    elif isinstance(content, (list, tuple)):
+        return [c(e, '{0}[{1}]'.format(json_path, i)) for i, e in enumerate(content)]
+    elif isinstance(content, dict):
+        return {k: c(v, '{0}[{1}]'.format(json_path, k))
+                for k, v in list(content.items())}
+    else:
+        param_type = type(content)
+        msg = 'Type {0} used for parameter {1} is not a number or a string' \
+            .format(param_type, json_path)
+        raise AirflowException(msg)
+
+
+def _handle_databricks_operator_execution(operator, context):
 
 Review comment:
   Can we explicitly pass the parameters: `hook` and `log`, into this function and lift the
logic to initially invoke `run-now` or `run-submit` back into the operator? The end result
would be a helper function responsible for polling for the run result.
   
   With this solution, we'd duplicate a bit of logic with the xcom business (which we can
refactor later) but we avoid having to pass the wide interface of an `operator` (which is
really a special databricks operato) to this function.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message