incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karthik Kambatla <kkamb...@cs.purdue.edu>
Subject Re: Support for a pub sub system
Date Fri, 09 Dec 2011 02:19:38 GMT
Well, that is very fair.

In addition to supporting pub/sub in the comm-layer, I think S4 can
actually be used to build a fault-tolerant, efficient content-based
pub/sub, but that is for a later date.

Thanks
Karthik

On Thu, Dec 8, 2011 at 9:02 PM, kishore g <g.kishore@gmail.com> wrote:

> External pub/sub is useful because they can provide better buffering what
> s4 system can provide. It is particularly helpful when the pipelines have
> different processing/computing times. Think of a case where you have a
> stage which is simply transforming data and another stage is actually doing
> some complex algorithm after accumulating some data. Very soon the second
> stage buffer will become full and start dropping events.
>
> It may not as efficient as doing a tcp/udp but has its own advantages. Some
> pub/sub also provides better fault tolerance.
>
> I am not advocating one over the other because it depends on the use case
> and what trade off user is willing to make. We should provide one such
> implementation to make sure we dont have something in s4 that makes it
> difficult to support this requirement.
>
> thanks,
> Kishore G
>
>
>
> On Thu, Dec 8, 2011 at 4:46 PM, Karthik Kambatla <kkambatl@cs.purdue.edu
> >wrote:
>
> > Hi Kishore
> >
> > Do you see any particular advantages of using pub/sub for inter-node
> > communication over UDP/TCP? The only advantage I see is an external
> pub/sub
> > being efficient than our implementations of UDP/TCP. If there is no
> > advantage, I don't think it makes any sense.
> >
> > Thanks
> > Karthik
> >
> > On Thu, Dec 8, 2011 at 7:26 PM, Leo Neumeyer <leoneumeyer@gmail.com>
> > wrote:
> >
> > > Sure, if  someone really wants it and is willing to design/implement,
> > > I don't see why not. It's a plugin so it wouldn't require major
> > > modifications.
> > >
> > > -leo
> > >
> > > On Thu, Dec 8, 2011 at 2:39 PM, kishore g <g.kishore@gmail.com> wrote:
> > > > Sorry, too many lists.
> > > >
> > > > Comm layer is supporting p2p. I remember some time back there was an
> > ask
> > > > for support for Active MQ in s4. The comm layer right now can support
> > > that
> > > > but we need to write one implementation to write/read to/from a pub
> sub
> > > to
> > > > make sure we can support it. We can do it for Active MQ. This is post
> > 0.5
> > > >
> > > > On Thu, Dec 8, 2011 at 2:25 PM, Leo Neumeyer <leoneumeyer@gmail.com>
> > > wrote:
> > > >
> > > >> Hi Kishore,
> > > >>
> > > >> Seems to me that our comm layer is already implementing a type of
> > > >> pub/sub system but with more granularity based on keys. I'm not very
> > > >> familiar with pub/sub systems though. Can you tell us more about the
> > > >> motivation? Is it to integrate with existing systems? In any case,
> you
> > > >> are not suggesting this for v0.5, right?
> > > >>
> > > >> (BTW, we should move this to s4-dev)
> > > >>
> > > >> -leo
> > > >>
> > > >> On Thu, Dec 8, 2011 at 1:02 PM, kishore g <g.kishore@gmail.com>
> > wrote:
> > > >> > Hi,
> > > >> >
> > > >> > In s4, the comm layer currently provides support for udp/tcp.
We
> > > should
> > > >> add
> > > >> > support for a pub/sub layer. There are already some pub/sub
> systems
> > > out
> > > >> > there. Any thoughts ?
> > > >> >
> > > >> > thanks,
> > > >> > Kishore G
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Leo Neumeyer (@leoneu)
> > > >>
> > >
> > >
> > >
> > > --
> > >
> > > Leo Neumeyer (@leoneu)
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message