incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Flavio Junqueira (JIRA)" <>
Subject [jira] [Commented] (S4-62) Multithreaded Streams
Date Wed, 11 Jul 2012 09:32:35 GMT


Flavio Junqueira commented on S4-62:

In ZooKeeper we do pipelining too, and it is essentially the stage-driven approach you're
referring to. The request processors would be the stages. 

I still feel that the case here is somewhat different. For ZooKeeper and I believe SEDA too,
we throttle when load is high, which halts the submission of new requests until the load drops.

We can't stop streams. If we throttle, then the end result will be dropping events, perhaps
many of them. Consequently, it does not sound great to have blocking I/O operations in the
critical path. It is unclear to me if adding thread pools would help because depending on
the arrival rate and the service rate, we may end up saturating the pool very fast.    
> Multithreaded Streams
> ---------------------
>                 Key: S4-62
>                 URL:
>             Project: Apache S4
>          Issue Type: Improvement
>    Affects Versions: 0.5
>            Reporter: Daniel Gómez Ferro
>         Attachments: S4-62-multithreaded-streams.patch, S4-62-multithreaded-streams.patch
> Currently, each Stream has one Thread in charge of processing the incoming events on
the appropriate PE. If one PE blocks it's execution while processing an event, the whole Stream
would be blocked. The current solution is for a PE to manage his own async thread, which I
don't think it's nice.
> It would be better to have a configurable number of threads that would take care of the
execution of the incoming events.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message