incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Junqueira <...@yahoo-inc.com>
Subject Re: Support for a pub sub system
Date Thu, 15 Dec 2011 16:56:34 GMT
I like the idea of using pub/sub. It could be used in cases that  
losing events is not acceptable. One system that we have some good  
knowledge about and could consider is Hedwig, since it is scalable and  
provides strong durability guarantees: http://zookeeper.apache.org/bookkeeper

-Flavio

On Dec 9, 2011, at 3:02 AM, kishore g wrote:

> External pub/sub is useful because they can provide better buffering  
> what
> s4 system can provide. It is particularly helpful when the pipelines  
> have
> different processing/computing times. Think of a case where you have a
> stage which is simply transforming data and another stage is  
> actually doing
> some complex algorithm after accumulating some data. Very soon the  
> second
> stage buffer will become full and start dropping events.
>
> It may not as efficient as doing a tcp/udp but has its own  
> advantages. Some
> pub/sub also provides better fault tolerance.
>
> I am not advocating one over the other because it depends on the use  
> case
> and what trade off user is willing to make. We should provide one such
> implementation to make sure we dont have something in s4 that makes it
> difficult to support this requirement.
>
> thanks,
> Kishore G
>
>
>
> On Thu, Dec 8, 2011 at 4:46 PM, Karthik Kambatla <kkambatl@cs.purdue.edu 
> >wrote:
>
>> Hi Kishore
>>
>> Do you see any particular advantages of using pub/sub for inter-node
>> communication over UDP/TCP? The only advantage I see is an external  
>> pub/sub
>> being efficient than our implementations of UDP/TCP. If there is no
>> advantage, I don't think it makes any sense.
>>
>> Thanks
>> Karthik
>>
>> On Thu, Dec 8, 2011 at 7:26 PM, Leo Neumeyer <leoneumeyer@gmail.com>
>> wrote:
>>
>>> Sure, if  someone really wants it and is willing to design/ 
>>> implement,
>>> I don't see why not. It's a plugin so it wouldn't require major
>>> modifications.
>>>
>>> -leo
>>>
>>> On Thu, Dec 8, 2011 at 2:39 PM, kishore g <g.kishore@gmail.com>  
>>> wrote:
>>>> Sorry, too many lists.
>>>>
>>>> Comm layer is supporting p2p. I remember some time back there was  
>>>> an
>> ask
>>>> for support for Active MQ in s4. The comm layer right now can  
>>>> support
>>> that
>>>> but we need to write one implementation to write/read to/from a  
>>>> pub sub
>>> to
>>>> make sure we can support it. We can do it for Active MQ. This is  
>>>> post
>> 0.5
>>>>
>>>> On Thu, Dec 8, 2011 at 2:25 PM, Leo Neumeyer  
>>>> <leoneumeyer@gmail.com>
>>> wrote:
>>>>
>>>>> Hi Kishore,
>>>>>
>>>>> Seems to me that our comm layer is already implementing a type of
>>>>> pub/sub system but with more granularity based on keys. I'm not  
>>>>> very
>>>>> familiar with pub/sub systems though. Can you tell us more about  
>>>>> the
>>>>> motivation? Is it to integrate with existing systems? In any  
>>>>> case, you
>>>>> are not suggesting this for v0.5, right?
>>>>>
>>>>> (BTW, we should move this to s4-dev)
>>>>>
>>>>> -leo
>>>>>
>>>>> On Thu, Dec 8, 2011 at 1:02 PM, kishore g <g.kishore@gmail.com>
>> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> In s4, the comm layer currently provides support for udp/tcp. We
>>> should
>>>>> add
>>>>>> support for a pub/sub layer. There are already some pub/sub  
>>>>>> systems
>>> out
>>>>>> there. Any thoughts ?
>>>>>>
>>>>>> thanks,
>>>>>> Kishore G
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Leo Neumeyer (@leoneu)
>>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Leo Neumeyer (@leoneu)
>>>
>>

flavio
junqueira

research scientist

fpj@yahoo-inc.com
direct +34 93-183-8828

avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message