incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nan N. Jiang" <>
Subject Re: [Discuss] Runtime Isolation for s4?
Date Fri, 11 May 2012 09:58:11 GMT
Yes. If there is no sharing or dependencies between apps, there is no need
to share the cluster.

But if there is sharing or dependencies between apps, for example: sharing
event stream(thinking about more then one apps are interested in the same
event stream case), how could s4 solve this issue with logical cluster?

-- Nan

On 5/8/12 9:30 PM, "Leo Neumeyer" <> wrote:

> Seems to me that the simplest solution is to create a logical cluster for
> each app or group of apps that need to be isolated. Using various smaller
> clusters may be more efficient that using a single large one because there
> fewer nodes use less network traffic and makes it easier to plan resource
> allocation. If there is no sharing or dependencies between apps, there is
> no need to share the cluster. The tools will make it easy to create
> clusters. Does this make sense for your use case?
> -leo
> On Tue, May 1, 2012 at 12:08 PM, Matthieu Morel <> wrote:
>> On 4/29/12 2:49 AM, Nan N. Jiang wrote:
>>> Hi Team,
>>> I want to start a discuss about ŒRuntime isolation¹ for s4.
>>> Currently, in S4 and S4-piper, following S4¹s symmetric principal, all
>>> applications will deployed in all nodes.
>>> And when starting s4 service, all applications deployed will run in the
>>> same JVM.
>>> This can be a disaster.
>>> Different application will fight for the JVM/hardware resource.
>>> Bad application even could kill other applications.
>>> Yes. Application Isolation could release the pain.
>>> But all applications are still fight for the resource in the same node.
>>> And all the PE in the same Application still fight for JVM resource.
>>> Thinking about a big application which contains hundreds of PE prototypes,
>>> If we want start this application in one JVM,
>>> Our s4 cluster must have a set of very high performance machines due to
>>> the CPU/Memory and other requirement of this application.
>>> This will limit the usage of S4 in big solution.
>>> I think the future of s4 cluster will be a big set of normal boxes
>>> instead of a small set of high performance boxes.
>>> So, compare application isolation, could we go further?
>>> Could we provide Runtime Isolation function for S4?
>>> Could we change the local deploy unit from application to a group of PE?
>>> even one PE?
>>> If this function can be provided by s4,
>>> Every PE could run in it¹s own JVM if needed.
>>> Then the resource fighting between PE will be minimum.
>> Deploying different PEs to different nodes would break the symmetry which
>> is a key property of the design of S4. However, you are pointing out a real
>> issue, and one way to provide PE isolation might be at the level of
>> partitioning. We could provide a way to provide exclusive partitioning for
>> some PEs using a partition exclusiveness scheme, which would be propagated
>> to every node. Concretely, messages for PE with key K (or a set of keys)
>> would be sent to node N, and only messages with such keys would be sent to
>> node N. This way, we provide PE isolation, without modifying the design of
>> the platform.
>> Do you think this could properly address the isolation issue?
>> To all: would that harm the overall design of the platform?
>> Thanks!
>> Matthieu
>>> Let¹s discuss about this.
>>> Regards,
>>> Jiang Nan

View raw message