incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Gómez Ferro (JIRA) <>
Subject [jira] [Commented] (S4-62) Multithreaded Streams
Date Fri, 22 Jun 2012 08:37:42 GMT


Daniel Gómez Ferro commented on S4-62:

My original use case has to do with waiting for I/O. Suppose you are receiving raw URLs and
you want to resolve them (in case they are shortened). You have to either resolve them from
your PE, use a separate service or query a database, all of them blocking your pipeline. 

As I said, the only alternative I could think of was starting an async Thread from the PE
to do this task, but I think this solution is much better.

If anyone knows a different solution please share. Maybe S4 is not the right tool for the
job, but overall I feel it matches nicely.
> Multithreaded Streams
> ---------------------
>                 Key: S4-62
>                 URL:
>             Project: Apache S4
>          Issue Type: Improvement
>            Reporter: Daniel Gómez Ferro
>         Attachments: S4-62-multithreaded-streams.patch
> Currently, each Stream has one Thread in charge of processing the incoming events on
the appropriate PE. If one PE blocks it's execution while processing an event, the whole Stream
would be blocked. The current solution is for a PE to manage his own async thread, which I
don't think it's nice.
> It would be better to have a configurable number of threads that would take care of the
execution of the incoming events.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message