incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nan N. Jiang" <jiang...@yahoo-inc.com>
Subject Re: [Discuss] Runtime Isolation for s4?
Date Mon, 28 May 2012 10:50:39 GMT
Hi Flavio,

I think in the ideal case, S4 should act as a real cluster.
We can deploy our APP in the S4 cluster and we don't need to think about how
and where the app is deployed.
The APP just need to specify which stream(event type) it's interested, then
if there is event generated for this stream, the event will flow to this
APP. S4 as a platform should help to handle this instead of using feeds
copies. 

With this feature, APP will be free from finding the way how to get the
incoming events and focus what kind of events it want.

-Nan 



On 5/25/12 1:50 PM, "Flavio Junqueira" <fpj@yahoo-inc.com> wrote:

> Nan, I'm not sure I understand what sharing means. Does it mean that each app
> gets a copy of the stream or you want each app to get a fraction of the
> stream? In any case, you seem to need a third app perhaps that feeds copies of
> the stream.
> 
> -Flavio
> 
> On May 11, 2012, at 11:58 AM, Nan N. Jiang wrote:
> 
>> Yes. If there is no sharing or dependencies between apps, there is no need
>> to share the cluster.
>> 
>> But if there is sharing or dependencies between apps, for example: sharing
>> event stream(thinking about more then one apps are interested in the same
>> event stream case), how could s4 solve this issue with logical cluster?
>> 
>> -- Nan
>> 
>> On 5/8/12 9:30 PM, "Leo Neumeyer" <leoneumeyer@gmail.com> wrote:
>> 
>>> Seems to me that the simplest solution is to create a logical cluster for
>>> each app or group of apps that need to be isolated. Using various smaller
>>> clusters may be more efficient that using a single large one because there
>>> fewer nodes use less network traffic and makes it easier to plan resource
>>> allocation. If there is no sharing or dependencies between apps, there is
>>> no need to share the cluster. The tools will make it easy to create
>>> clusters. Does this make sense for your use case?
>>> 
>>> -leo
>>> 
>>> On Tue, May 1, 2012 at 12:08 PM, Matthieu Morel <mmorel@apache.org> wrote:
>>> 
>>>> On 4/29/12 2:49 AM, Nan N. Jiang wrote:
>>>> 
>>>>> Hi Team,
>>>>> 
>>>>> I want to start a discuss about ŒRuntime isolation¹ for s4.
>>>>> 
>>>>> Currently, in S4 and S4-piper, following S4¹s symmetric principal, all
>>>>> applications will deployed in all nodes.
>>>>> And when starting s4 service, all applications deployed will run in the
>>>>> same JVM.
>>>>> This can be a disaster.
>>>>> Different application will fight for the JVM/hardware resource.
>>>>> Bad application even could kill other applications.
>>>>> 
>>>>> Yes. Application Isolation could release the pain.
>>>>> But all applications are still fight for the resource in the same node.
>>>>> And all the PE in the same Application still fight for JVM resource.
>>>>> 
>>>>> Thinking about a big application which contains hundreds of PE prototypes,
>>>>> If we want start this application in one JVM,
>>>>> Our s4 cluster must have a set of very high performance machines due
to
>>>>> the CPU/Memory and other requirement of this application.
>>>>> This will limit the usage of S4 in big solution.
>>>>> I think the future of s4 cluster will be a big set of normal boxes
>>>>> instead of a small set of high performance boxes.
>>>>> 
>>>>> So, compare application isolation, could we go further?
>>>>> Could we provide Runtime Isolation function for S4?
>>>>> Could we change the local deploy unit from application to a group of
PE?
>>>>> even one PE?
>>>>> If this function can be provided by s4,
>>>>> Every PE could run in it¹s own JVM if needed.
>>>>> Then the resource fighting between PE will be minimum.
>>>>> 
>>>> 
>>>> Deploying different PEs to different nodes would break the symmetry which
>>>> is a key property of the design of S4. However, you are pointing out a real
>>>> issue, and one way to provide PE isolation might be at the level of
>>>> partitioning. We could provide a way to provide exclusive partitioning for
>>>> some PEs using a partition exclusiveness scheme, which would be propagated
>>>> to every node. Concretely, messages for PE with key K (or a set of keys)
>>>> would be sent to node N, and only messages with such keys would be sent to
>>>> node N. This way, we provide PE isolation, without modifying the design of
>>>> the platform.
>>>> 
>>>> Do you think this could properly address the isolation issue?
>>>> To all: would that harm the overall design of the platform?
>>>> 
>>>> Thanks!
>>>> 
>>>> Matthieu
>>>> 
>>>> 
>>>> 
>>>>> Let¹s discuss about this.
>>>>> 
>>>>> Regards,
>>>>> Jiang Nan
>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 
> flavio
> junqueira
> senior research scientist
>  
> fpj@yahoo-inc.com
> direct +34 93-183-8828
>  
> avinguda diagonal 177, 8th floor, barcelona, 08018, es
> phone (408) 349 3300    fax (408) 349 3301
> 

Mime
View raw message