spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vanzin <>
Subject [GitHub] spark issue #16291: [SPARK-18838][WIP] Use separate executor service for eac...
Date Thu, 15 Dec 2016 19:30:29 GMT
Github user vanzin commented on the issue:
    Hmm... I took a quick look and I'm not sure I understand exactly what's going on. It seems
you're wrapping each listener with a `ListenerEvenProcessor` (note the typo), and each processor
has its own thread pool for processing events. 
    If that's the case, that sounds wrong. Each listener should process events serially otherwise
you risk getting into funny situations like a task end event being processed before the task
start event for the same task.
    I think this would benefit from a proper explanation of the changes being proposed, instead
of a bunch of code. How will the listener bus work with the changes? Where will we have thread
pools, where will we have message queues? Will each listener get its own dedicated thread,
or will there be a limit? What kind of controls will there be to avoid memory pressure? Would
it be worth it to allow certain listeners to have dedicated threads while others share the
same one like the current approach? That kind of thing. Can you write something up and post
it to the bug?

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message