incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Gómez Ferro (JIRA) <>
Subject [jira] [Commented] (S4-62) Multithreaded Streams
Date Fri, 22 Jun 2012 14:31:42 GMT


Daniel Gómez Ferro commented on S4-62:

bq. We could have a thread pool on the side just to deal with such requests, but this would
be effective only if not all events require blocking on I/O.

Matthieu and I also discussed this and realized that it would be equivalent to having a PE
on the side doing such requests. You could split the incoming stream into one going to a PE
doing blocking requests and another stream to a non-blocking PE, effectively moving the I/O
out of the critical path.

bq. If anyone has more thoughts on how to deal with such cases, I would be interested in hearing.
This discussion goes beyond one vs. multiple threads.

Yes, it would be really nice to know if anybody has run into this kind of issue while developing
S4 applications. How did you avoid it?
> Multithreaded Streams
> ---------------------
>                 Key: S4-62
>                 URL:
>             Project: Apache S4
>          Issue Type: Improvement
>    Affects Versions: 0.5
>            Reporter: Daniel Gómez Ferro
>         Attachments: S4-62-multithreaded-streams.patch, S4-62-multithreaded-streams.patch
> Currently, each Stream has one Thread in charge of processing the incoming events on
the appropriate PE. If one PE blocks it's execution while processing an event, the whole Stream
would be blocked. The current solution is for a PE to manage his own async thread, which I
don't think it's nice.
> It would be better to have a configurable number of threads that would take care of the
execution of the incoming events.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message