incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Junqueira <...@yahoo-inc.com>
Subject Re: [Discuss] Runtime Isolation for s4?
Date Tue, 29 May 2012 17:33:30 GMT
Hi Nan, In my understanding, the design of piper supports exactly the case you're saying. You
can have a producer app that produces a stream and consumer apps subscribe to it. My point
was simply that you need a third app to be the producer of the stream. The consumers of the
stream do not need to be aware of how the stream is being produced. 

-Flavio

On May 28, 2012, at 12:50 PM, Nan N. Jiang wrote:

> Hi Flavio,
> 
> I think in the ideal case, S4 should act as a real cluster.
> We can deploy our APP in the S4 cluster and we don't need to think about how
> and where the app is deployed.
> The APP just need to specify which stream(event type) it's interested, then
> if there is event generated for this stream, the event will flow to this
> APP. S4 as a platform should help to handle this instead of using feeds
> copies. 
> 
> With this feature, APP will be free from finding the way how to get the
> incoming events and focus what kind of events it want.
> 
> -Nan 
> 
> 
> 
> On 5/25/12 1:50 PM, "Flavio Junqueira" <fpj@yahoo-inc.com> wrote:
> 
>> Nan, I'm not sure I understand what sharing means. Does it mean that each app
>> gets a copy of the stream or you want each app to get a fraction of the
>> stream? In any case, you seem to need a third app perhaps that feeds copies of
>> the stream.
>> 
>> -Flavio
>> 
>> On May 11, 2012, at 11:58 AM, Nan N. Jiang wrote:
>> 
>>> Yes. If there is no sharing or dependencies between apps, there is no need
>>> to share the cluster.
>>> 
>>> But if there is sharing or dependencies between apps, for example: sharing
>>> event stream(thinking about more then one apps are interested in the same
>>> event stream case), how could s4 solve this issue with logical cluster?
>>> 
>>> -- Nan
>>> 
>>> On 5/8/12 9:30 PM, "Leo Neumeyer" <leoneumeyer@gmail.com> wrote:
>>> 
>>>> Seems to me that the simplest solution is to create a logical cluster for
>>>> each app or group of apps that need to be isolated. Using various smaller
>>>> clusters may be more efficient that using a single large one because there
>>>> fewer nodes use less network traffic and makes it easier to plan resource
>>>> allocation. If there is no sharing or dependencies between apps, there is
>>>> no need to share the cluster. The tools will make it easy to create
>>>> clusters. Does this make sense for your use case?
>>>> 
>>>> -leo
>>>> 
>>>> On Tue, May 1, 2012 at 12:08 PM, Matthieu Morel <mmorel@apache.org>
wrote:
>>>> 
>>>>> On 4/29/12 2:49 AM, Nan N. Jiang wrote:
>>>>> 
>>>>>> Hi Team,
>>>>>> 
>>>>>> I want to start a discuss about ŒRuntime isolation¹ for s4.
>>>>>> 
>>>>>> Currently, in S4 and S4-piper, following S4¹s symmetric principal,
all
>>>>>> applications will deployed in all nodes.
>>>>>> And when starting s4 service, all applications deployed will run
in the
>>>>>> same JVM.
>>>>>> This can be a disaster.
>>>>>> Different application will fight for the JVM/hardware resource.
>>>>>> Bad application even could kill other applications.
>>>>>> 
>>>>>> Yes. Application Isolation could release the pain.
>>>>>> But all applications are still fight for the resource in the same
node.
>>>>>> And all the PE in the same Application still fight for JVM resource.
>>>>>> 
>>>>>> Thinking about a big application which contains hundreds of PE prototypes,
>>>>>> If we want start this application in one JVM,
>>>>>> Our s4 cluster must have a set of very high performance machines
due to
>>>>>> the CPU/Memory and other requirement of this application.
>>>>>> This will limit the usage of S4 in big solution.
>>>>>> I think the future of s4 cluster will be a big set of normal boxes
>>>>>> instead of a small set of high performance boxes.
>>>>>> 
>>>>>> So, compare application isolation, could we go further?
>>>>>> Could we provide Runtime Isolation function for S4?
>>>>>> Could we change the local deploy unit from application to a group
of PE?
>>>>>> even one PE?
>>>>>> If this function can be provided by s4,
>>>>>> Every PE could run in it¹s own JVM if needed.
>>>>>> Then the resource fighting between PE will be minimum.
>>>>>> 
>>>>> 
>>>>> Deploying different PEs to different nodes would break the symmetry which
>>>>> is a key property of the design of S4. However, you are pointing out
a real
>>>>> issue, and one way to provide PE isolation might be at the level of
>>>>> partitioning. We could provide a way to provide exclusive partitioning
for
>>>>> some PEs using a partition exclusiveness scheme, which would be propagated
>>>>> to every node. Concretely, messages for PE with key K (or a set of keys)
>>>>> would be sent to node N, and only messages with such keys would be sent
to
>>>>> node N. This way, we provide PE isolation, without modifying the design
of
>>>>> the platform.
>>>>> 
>>>>> Do you think this could properly address the isolation issue?
>>>>> To all: would that harm the overall design of the platform?
>>>>> 
>>>>> Thanks!
>>>>> 
>>>>> Matthieu
>>>>> 
>>>>> 
>>>>> 
>>>>>> Let¹s discuss about this.
>>>>>> 
>>>>>> Regards,
>>>>>> Jiang Nan
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> flavio
>> junqueira
>> senior research scientist
>> 
>> fpj@yahoo-inc.com
>> direct +34 93-183-8828
>> 
>> avinguda diagonal 177, 8th floor, barcelona, 08018, es
>> phone (408) 349 3300    fax (408) 349 3301
>> 
> 

flavio
junqueira
senior research scientist
 
fpj@yahoo-inc.com
direct +34 93-183-8828
 
avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message