airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bo...@apache.org
Subject incubator-airflow git commit: [AIRFLOW-654] Add SSL Config Option for CeleryExecutor w/ RabbitMQ - Add BROKER_USE_SSL config to give option to send AMQP messages over SSL - Can be set using usual airflow options (e.g. airflow.cfg, env vars, etc.)
Date Thu, 01 Jun 2017 08:19:36 GMT
Repository: incubator-airflow
Updated Branches:
  refs/heads/master 71d8f132c -> 868bfe4ca


[AIRFLOW-654] Add SSL Config Option for CeleryExecutor w/ RabbitMQ
- Add BROKER_USE_SSL config to give option to send AMQP messages over SSL
- Can be set using usual airflow options (e.g. airflow.cfg, env vars, etc.)

Closes #2333 from forsberg/ssl_amqp


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/868bfe4c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/868bfe4c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/868bfe4c

Branch: refs/heads/master
Commit: 868bfe4cab91e306f450b8560915918351af341c
Parents: 71d8f13
Author: Michael Otte <michaelotte1@gmail.com>
Authored: Thu Jun 1 10:19:30 2017 +0200
Committer: Bolke de Bruin <bolke@xs4all.nl>
Committed: Thu Jun 1 10:19:30 2017 +0200

----------------------------------------------------------------------
 airflow/executors/celery_executor.py | 13 +++++++++++++
 docs/security.rst                    | 11 +++++++++++
 2 files changed, 24 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/868bfe4c/airflow/executors/celery_executor.py
----------------------------------------------------------------------
diff --git a/airflow/executors/celery_executor.py b/airflow/executors/celery_executor.py
index e0c94c1..4de8252 100644
--- a/airflow/executors/celery_executor.py
+++ b/airflow/executors/celery_executor.py
@@ -15,6 +15,7 @@
 from builtins import object
 import logging
 import subprocess
+import ssl
 import time
 
 from celery import Celery
@@ -46,6 +47,18 @@ class CeleryConfig(object):
     CELERYD_CONCURRENCY = configuration.getint('celery', 'CELERYD_CONCURRENCY')
     CELERY_DEFAULT_QUEUE = DEFAULT_QUEUE
     CELERY_DEFAULT_EXCHANGE = DEFAULT_QUEUE
+    if configuration.get('celery', 'CELERY_SSL_ACTIVE'):
+        try:
+            BROKER_USE_SSL = {'keyfile': configuration.get('celery', 'CELERY_SSL_KEY'),
+                              'certfile': configuration.get('celery', 'CELERY_SSL_CERT'),
+                              'ca_certs': configuration.get('celery', 'CELERY_SSL_CACERT'),
+                              'cert_reqs': ssl.CERT_REQUIRED}
+        except ValueError:
+            raise AirflowException('ValueError: CELERY_SSL_ACTIVE is True, please ensure
CELERY_SSL_KEY, '
+                                   'CELERY_SSL_CERT and CELERY_SSL_CACERT are set')
+        except Exception as e:
+            raise AirflowException('Exception: There was an unknown Celery SSL Error.  Please
ensure you want to use '
+                                   'SSL and/or have all necessary certs and key.')
 
 app = Celery(
     configuration.get('celery', 'CELERY_APP_NAME'),

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/868bfe4c/docs/security.rst
----------------------------------------------------------------------
diff --git a/docs/security.rst b/docs/security.rst
index ada34a2..6c0893d 100644
--- a/docs/security.rst
+++ b/docs/security.rst
@@ -321,6 +321,17 @@ standard port 443, you'll need to configure that too. Be aware that super
user p
     web_server_port = 443
     base_url = http://<hostname or IP>:443
 
+Enable CeleryExecutor with SSL. Ensure you properly generate client and server
+certs and keys.
+
+.. code-block:: bash
+
+    [celery]
+    CELERY_SSL_ACTIVE = True
+    CELERY_SSL_KEY = <path to key>
+    CELERY_SSL_CERT = <path to cert>
+    CELERY_SSL_CACERT = <path to cacert>
+
 Impersonation
 -------------
 


Mime
View raw message