incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leo Neumeyer <leoneume...@gmail.com>
Subject Re: hangout notes
Date Tue, 27 Mar 2012 23:22:16 GMT
Not sure it would make much difference. In the current design we can have
several apps running in the same cluster. One strange thing is that to run
several apps on a single server, you would still create a subcluster for
each. I suppose that as long as the user doesn't see it, it doesn't really
matter.

-leo

On Tue, Mar 27, 2012 at 4:02 PM, Karthik Kambatla <kkambatl@cs.purdue.edu>wrote:

> I didn't realize I was proposing something new, that was just my
> understanding of your idea :)
>
> If there is no downside to have one subcluster per app, having app-specific
> subclusters would really help with the routing.
>
> Thanks
> Karthik
>
> On Tue, Mar 27, 2012 at 6:40 PM, Leo Neumeyer <leoneumeyer@gmail.com>
> wrote:
>
> > I'm OK with finding a better nomenclature but you seem to be suggesting a
> > different, and perhaps better, idea.
> >
> > In my original thinking nodes I use the following logical statements:
> >
> > - A cluster has M subclusters
> > - A subcluster id=i has Ni nodes
> > - There are a total of sum{i=0..M-1}(Ni) nodes
> > - A node is part of one and only one subcluster
> > - An app instance is activated in one and only one subcluster
> > - A subcluster can have several app instances (there is no one-to-one
> > relationship between subcluster and apps)
> >
> > Base on Karthik comments, it sounds like he is proposing to create a
> > subcluster for each app instance, that is the subcluster only exists for
> a
> > specific app instance. I hadn't thought about that but it might not be a
> > bad idea.
> >
> > In that case a cluster is a collection of subclusters where each
> subcluster
> > is created and destroyed when an app is created and destroyed. I kind of
> > like this, is there a downside? For example, to run a query to analyze RT
> > data, we may want to deploy an App. Would the overhead of creating a
> > subcluster every time be a problem compared to deploying to an existing
> > logical subcluster?
> >
> > Thoughts??
> >
> > -leo
> >
> > On Tue, Mar 27, 2012 at 3:03 PM, Karthik Kambatla <
> kkambatl@cs.purdue.edu
> > >wrote:
> >
> > > Indeed the demo was significant progress, now we can actually start
> > > thinking of releasing the piper.
> > >
> > > The notion of "Cluster" and "Sub-cluster" makes absolute sense, but the
> > > nomenclature is confusing. Can we name them differently - for instance,
> > S4
> > > (physical) Cluster and S4 App-Cluster, or something completely
> different?
> > > Each sub-cluster (app-cluster) can be identified by the App ID. We can
> > use
> > > AppID for routing to a sub-cluster (app-cluster).
> > >
> > > Thanks
> > > Karthik
> > >
> > > On Tue, Mar 27, 2012 at 1:18 PM, Leo Neumeyer <leoneumeyer@gmail.com>
> > > wrote:
> > >
> > > > Matthieu,
> > > >
> > > > Congrats again for the great demo using S4 across subclusters and the
> > > > PlayFramework-like command line tool to manage the cluster and deploy
> > > apps.
> > > > We are basically providing a full stack solution so users don't have
> to
> > > > spend time getting systems to work together.
> > > >
> > > > Here is a summary of today's comments. Please let me know if you
> agree
> > or
> > > > if you propose a different approach. Let's make sure we are all in
> > sync.
> > > >
> > > >  * We need to be able to extract routing info from messages and
> > > > deserialize the Event at a higher level where we know the Event
> Class.
> > > >
> > > >  * Definition of "S4 Cluster": a collection of S4 sub-clusters.
> > > >
> > > >  * Definition of "S4 Sub-cluster": A collection of nodes (each
> > > sub-cluster
> > > > has its own partitions).
> > > >
> > > >  * All nodes in an "S4 Cluster" are symmetric (same code image in ALL
> > > > nodes across sub-clusters.)
> > > >
> > > >  * For total isolation, send event stream to another "S4 Cluster".
> > > >
> > > >  * A specific app is deployed to ALL nodes in the "S4 Cluster" but
> > > > activated on a specific "S4 Sub-cluster". This provides flexibility
> to
> > > > assign resources to groups of Apps while maintaining simplicity via
> > local
> > > > method calls. (Remote invokation is also handled by the platform in
> the
> > > > same way and transparently.)
> > > >
> > > >  * Each Node has a local collection of Sender instances, one for for
> > each
> > > > sub-cluster in the cluster.
> > > >
> > > >  * When sending an Event to an App, we use runtime tables to lookup
> the
> > > > sub-cluster and the corresponding Sender. Once we know the Sender, we
> > > > simply use it to send the Event using local method calls. All the
> > > > complexity is abstracted out.
> > > >
> > > >  * Use EventSource to avoid application dependencies. Apps subscribe
> to
> > > > data sources. The event source is looked up when an App is loaded
> using
> > > > runtime tables.
> > > >
> > > > There are additional comments in the Jira.
> > > >
> > > > If you are reading this and would like to contribute to the project,
> > > please
> > > > email us! Let us know if you would like to join our weekly Google
> > hangout
> > > > where we discuss the project.
> > > >
> > > > --
> > > >
> > > > Leo Neumeyer (@leoneu)
> > > >
> > >
> >
> >
> >
> > --
> >
> > Leo Neumeyer (@leoneu)
> >
>



-- 

Leo Neumeyer (@leoneu)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message