spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Gao <todd.gao.2013+sp...@gmail.com>
Subject CallbackServer in PySpark Streaming
Date Wed, 11 Feb 2015 08:32:49 GMT
Hi all,

I am reading the code of PySpark and its Streaming module.

In PySpark Streaming, when the `compute` method of the instance of
PythonTransformedDStream is invoked, a connection to the CallbackServer
is created internally.
I wonder where is the CallbackServer for each PythonTransformedDStream
instance on the slave nodes in distributed environment.
Is there a CallbackServer running on every slave node?

thanks
Todd

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message