incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthieu Morel (JIRA)" <>
Subject [jira] [Commented] (S4-62) Multithreaded Streams
Date Fri, 22 Jun 2012 09:04:42 GMT


Matthieu Morel commented on S4-62:

Ideally, there should not be any blocking operations in PEs. However, this only applies if
you don't need any external data, or that you can load all external data in memory upon initialization.
In some cases, this is not possible, and  I/O will be needed. Retrieved data can then be cached.
It's much better if the stream is not blocked (provided you don't need ordering), and we certainly
don't want to let PEs manage threads internally (that would mess the lifecycle management
of PEs).

In addition, parallelizing the processing of streams may also help to increase throughput,
and it is something we had been contemplating, but that we preferred to focus on in a later
release. Daniel's proposal is a good start, with reasonable defaults, and remains optional.
We could later improve this mechanism and provide more flexibility / performance.
> Multithreaded Streams
> ---------------------
>                 Key: S4-62
>                 URL:
>             Project: Apache S4
>          Issue Type: Improvement
>            Reporter: Daniel Gómez Ferro
>         Attachments: S4-62-multithreaded-streams.patch
> Currently, each Stream has one Thread in charge of processing the incoming events on
the appropriate PE. If one PE blocks it's execution while processing an event, the whole Stream
would be blocked. The current solution is for a PE to manage his own async thread, which I
don't think it's nice.
> It would be better to have a configurable number of threads that would take care of the
execution of the incoming events.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message