airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] feng-tao commented on a change in pull request #3830: [AIRFLOW-2156] Parallelize Celery Executor
Date Tue, 04 Sep 2018 23:01:45 GMT
feng-tao commented on a change in pull request #3830: [AIRFLOW-2156] Parallelize Celery Executor
URL: https://github.com/apache/incubator-airflow/pull/3830#discussion_r215090670
 
 

 ##########
 File path: airflow/executors/celery_executor.py
 ##########
 @@ -63,6 +69,40 @@ def execute_command(command):
         raise AirflowException('Celery command failed')
 
 
+class ExceptionWithTraceback(object):
+    """
+    Wrapper class used to propogate exceptions to parent processes from subprocesses.
+    :param exception: The exception to wrap
+    :type exception: Exception
+    :param traceback: The stacktrace to wrap
+    :type traceback: str
+    """
+
+    def __init__(self, exception, exception_traceback):
+        self.exception = exception
+        self.traceback = exception_traceback
+
+
+def fetch_celery_task_state(celery_task):
 
 Review comment:
   does the network call happen here? Based on http://docs.celeryproject.org/en/latest/reference/celery.result.html,
it uses celery_task.collect to fetch the result. I don't quite get the how the network call
happens in this func though.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message