incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Junqueira <>
Subject Re: [Discuss] Runtime Isolation for s4?
Date Tue, 29 May 2012 17:33:30 GMT
Hi Nan, In my understanding, the design of piper supports exactly the case you're saying. You
can have a producer app that produces a stream and consumer apps subscribe to it. My point
was simply that you need a third app to be the producer of the stream. The consumers of the
stream do not need to be aware of how the stream is being produced. 


On May 28, 2012, at 12:50 PM, Nan N. Jiang wrote:

> Hi Flavio,
> I think in the ideal case, S4 should act as a real cluster.
> We can deploy our APP in the S4 cluster and we don't need to think about how
> and where the app is deployed.
> The APP just need to specify which stream(event type) it's interested, then
> if there is event generated for this stream, the event will flow to this
> APP. S4 as a platform should help to handle this instead of using feeds
> copies. 
> With this feature, APP will be free from finding the way how to get the
> incoming events and focus what kind of events it want.
> -Nan 
> On 5/25/12 1:50 PM, "Flavio Junqueira" <> wrote:
>> Nan, I'm not sure I understand what sharing means. Does it mean that each app
>> gets a copy of the stream or you want each app to get a fraction of the
>> stream? In any case, you seem to need a third app perhaps that feeds copies of
>> the stream.
>> -Flavio
>> On May 11, 2012, at 11:58 AM, Nan N. Jiang wrote:
>>> Yes. If there is no sharing or dependencies between apps, there is no need
>>> to share the cluster.
>>> But if there is sharing or dependencies between apps, for example: sharing
>>> event stream(thinking about more then one apps are interested in the same
>>> event stream case), how could s4 solve this issue with logical cluster?
>>> -- Nan
>>> On 5/8/12 9:30 PM, "Leo Neumeyer" <> wrote:
>>>> Seems to me that the simplest solution is to create a logical cluster for
>>>> each app or group of apps that need to be isolated. Using various smaller
>>>> clusters may be more efficient that using a single large one because there
>>>> fewer nodes use less network traffic and makes it easier to plan resource
>>>> allocation. If there is no sharing or dependencies between apps, there is
>>>> no need to share the cluster. The tools will make it easy to create
>>>> clusters. Does this make sense for your use case?
>>>> -leo
>>>> On Tue, May 1, 2012 at 12:08 PM, Matthieu Morel <>
>>>>> On 4/29/12 2:49 AM, Nan N. Jiang wrote:
>>>>>> Hi Team,
>>>>>> I want to start a discuss about ŒRuntime isolation¹ for s4.
>>>>>> Currently, in S4 and S4-piper, following S4¹s symmetric principal,
>>>>>> applications will deployed in all nodes.
>>>>>> And when starting s4 service, all applications deployed will run
in the
>>>>>> same JVM.
>>>>>> This can be a disaster.
>>>>>> Different application will fight for the JVM/hardware resource.
>>>>>> Bad application even could kill other applications.
>>>>>> Yes. Application Isolation could release the pain.
>>>>>> But all applications are still fight for the resource in the same
>>>>>> And all the PE in the same Application still fight for JVM resource.
>>>>>> Thinking about a big application which contains hundreds of PE prototypes,
>>>>>> If we want start this application in one JVM,
>>>>>> Our s4 cluster must have a set of very high performance machines
due to
>>>>>> the CPU/Memory and other requirement of this application.
>>>>>> This will limit the usage of S4 in big solution.
>>>>>> I think the future of s4 cluster will be a big set of normal boxes
>>>>>> instead of a small set of high performance boxes.
>>>>>> So, compare application isolation, could we go further?
>>>>>> Could we provide Runtime Isolation function for S4?
>>>>>> Could we change the local deploy unit from application to a group
of PE?
>>>>>> even one PE?
>>>>>> If this function can be provided by s4,
>>>>>> Every PE could run in it¹s own JVM if needed.
>>>>>> Then the resource fighting between PE will be minimum.
>>>>> Deploying different PEs to different nodes would break the symmetry which
>>>>> is a key property of the design of S4. However, you are pointing out
a real
>>>>> issue, and one way to provide PE isolation might be at the level of
>>>>> partitioning. We could provide a way to provide exclusive partitioning
>>>>> some PEs using a partition exclusiveness scheme, which would be propagated
>>>>> to every node. Concretely, messages for PE with key K (or a set of keys)
>>>>> would be sent to node N, and only messages with such keys would be sent
>>>>> node N. This way, we provide PE isolation, without modifying the design
>>>>> the platform.
>>>>> Do you think this could properly address the isolation issue?
>>>>> To all: would that harm the overall design of the platform?
>>>>> Thanks!
>>>>> Matthieu
>>>>>> Let¹s discuss about this.
>>>>>> Regards,
>>>>>> Jiang Nan
>> flavio
>> junqueira
>> senior research scientist
>> direct +34 93-183-8828
>> avinguda diagonal 177, 8th floor, barcelona, 08018, es
>> phone (408) 349 3300    fax (408) 349 3301

senior research scientist
direct +34 93-183-8828
avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message