incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Flavio Junqueira (JIRA)" <>
Subject [jira] [Commented] (S4-62) Multithreaded Streams
Date Fri, 22 Jun 2012 13:53:42 GMT


Flavio Junqueira commented on S4-62:

I just had a chat with Daniel about this. My first reaction here was to think that PEs shouldn't
be blocking on I/O because you want to keep the processing of streams flowing. After seeing
the use case, I realized that it is valid one and there could be more like that. 

I was then wondering what the right way of dealing with such cases is. One high-level idea
is to move the processing of I/O requests out of the critical path so that we can keep processing
events. We could have a thread pool on the side just to deal with such requests, but this
would be effective only if not all events require blocking on I/O.

If anyone has more thoughts on how to deal with such cases, I would be interested in hearing.
This discussion goes beyond one vs. multiple threads.
> Multithreaded Streams
> ---------------------
>                 Key: S4-62
>                 URL:
>             Project: Apache S4
>          Issue Type: Improvement
>    Affects Versions: 0.5
>            Reporter: Daniel Gómez Ferro
>         Attachments: S4-62-multithreaded-streams.patch, S4-62-multithreaded-streams.patch
> Currently, each Stream has one Thread in charge of processing the incoming events on
the appropriate PE. If one PE blocks it's execution while processing an event, the whole Stream
would be blocked. The current solution is for a PE to manage his own async thread, which I
don't think it's nice.
> It would be better to have a configurable number of threads that would take care of the
execution of the incoming events.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message