incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: Support for a pub sub system
Date Fri, 09 Dec 2011 02:02:48 GMT
External pub/sub is useful because they can provide better buffering what
s4 system can provide. It is particularly helpful when the pipelines have
different processing/computing times. Think of a case where you have a
stage which is simply transforming data and another stage is actually doing
some complex algorithm after accumulating some data. Very soon the second
stage buffer will become full and start dropping events.

It may not as efficient as doing a tcp/udp but has its own advantages. Some
pub/sub also provides better fault tolerance.

I am not advocating one over the other because it depends on the use case
and what trade off user is willing to make. We should provide one such
implementation to make sure we dont have something in s4 that makes it
difficult to support this requirement.

thanks,
Kishore G



On Thu, Dec 8, 2011 at 4:46 PM, Karthik Kambatla <kkambatl@cs.purdue.edu>wrote:

> Hi Kishore
>
> Do you see any particular advantages of using pub/sub for inter-node
> communication over UDP/TCP? The only advantage I see is an external pub/sub
> being efficient than our implementations of UDP/TCP. If there is no
> advantage, I don't think it makes any sense.
>
> Thanks
> Karthik
>
> On Thu, Dec 8, 2011 at 7:26 PM, Leo Neumeyer <leoneumeyer@gmail.com>
> wrote:
>
> > Sure, if  someone really wants it and is willing to design/implement,
> > I don't see why not. It's a plugin so it wouldn't require major
> > modifications.
> >
> > -leo
> >
> > On Thu, Dec 8, 2011 at 2:39 PM, kishore g <g.kishore@gmail.com> wrote:
> > > Sorry, too many lists.
> > >
> > > Comm layer is supporting p2p. I remember some time back there was an
> ask
> > > for support for Active MQ in s4. The comm layer right now can support
> > that
> > > but we need to write one implementation to write/read to/from a pub sub
> > to
> > > make sure we can support it. We can do it for Active MQ. This is post
> 0.5
> > >
> > > On Thu, Dec 8, 2011 at 2:25 PM, Leo Neumeyer <leoneumeyer@gmail.com>
> > wrote:
> > >
> > >> Hi Kishore,
> > >>
> > >> Seems to me that our comm layer is already implementing a type of
> > >> pub/sub system but with more granularity based on keys. I'm not very
> > >> familiar with pub/sub systems though. Can you tell us more about the
> > >> motivation? Is it to integrate with existing systems? In any case, you
> > >> are not suggesting this for v0.5, right?
> > >>
> > >> (BTW, we should move this to s4-dev)
> > >>
> > >> -leo
> > >>
> > >> On Thu, Dec 8, 2011 at 1:02 PM, kishore g <g.kishore@gmail.com>
> wrote:
> > >> > Hi,
> > >> >
> > >> > In s4, the comm layer currently provides support for udp/tcp. We
> > should
> > >> add
> > >> > support for a pub/sub layer. There are already some pub/sub systems
> > out
> > >> > there. Any thoughts ?
> > >> >
> > >> > thanks,
> > >> > Kishore G
> > >>
> > >>
> > >>
> > >> --
> > >>
> > >> Leo Neumeyer (@leoneu)
> > >>
> >
> >
> >
> > --
> >
> > Leo Neumeyer (@leoneu)
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message