spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Armbrust (JIRA)" <>
Subject [jira] [Updated] (SPARK-11095) Simplify Netty RPC implementation by using a separate thread pool for each endpoint
Date Wed, 18 Nov 2015 22:13:11 GMT


Michael Armbrust updated SPARK-11095:
    Target Version/s: 2.0.0  (was: 1.6.0)

> Simplify Netty RPC implementation by using a separate thread pool for each endpoint
> -----------------------------------------------------------------------------------
>                 Key: SPARK-11095
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Spark Core
>            Reporter: Reynold Xin
>            Assignee: Shixiong Zhu
> The dispatcher class and the inbox class of the current Netty-based RPC implementation
is fairly complicated. It uses a single, shared thread pool to execute all the endpoints.
This is similar to how Akka does actor message dispatching. The benefit of this design is
that this RPC implementation can support a very large number of endpoints, as they are all
multiplexed into a single thread pool for execution. The downside is the complexity resulting
from synchronization and coordination.
> An alternative implementation is to have a separate message queue and thread pool for
each endpoint. The dispatcher simply routes the messages to the appropriate message queue,
and the threads poll the queue for messages to process.
> If the endpoint is single threaded, then the thread pool should contain only a single
thread. If the endpoint supports concurrent execution, then the thread pool should contain
more threads.
> Two additional things we need to be careful with are:
> 1. An endpoint should only process normal messages after OnStart is called. This can
be done by having the thread that starts the endpoint processing OnStart.
> 2. An endpoint should process OnStop after all normal messages have been processed. I
think this can be done by having a busy loop to spin until the size of the message queue is

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message