distributedlog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Junqueira <...@apache.org>
Subject Re: hundreds of millions of streams?
Date Sat, 29 Oct 2016 09:33:03 GMT
Perhaps this is an obvious observation, but can't you deploy multiple instances to scale to
whatever size you like?

-Flavio

> On 28 Oct 2016, at 20:27, Poule Dodue <pouledodue@hotmail.com> wrote:
> 
> I think it would cause more concurrency problems in write operations as
> described at page 12 point #2 of this thesis:
> 
> http://www.diva-portal.se/smash/get/diva2:877307/FULLTEXT01.pdf
> 
> 
> 
>> Le 28 oct. 2016 à 11:48, Leigh Stewart <lstewart@twitter.com.INVALID> a écrit
:
>> 
>> Got it. We probably can't support that scale at this time.
>> Curious: do you resort to sharing streams among objects with systems that
>> don't support 100s millions of streams? (i.e. partitioning objects across
>> streams?)
>> 
>> On Fri, Oct 28, 2016 at 8:24 AM, Poule Dodue <pouledodue@hotmail.com> wrote:
>> 
>>> yes aka ES/CQRS
>>> 
>>> some links:
>>> 
>>> https://msdn.microsoft.com/en-us/library/jj554200.aspx
>>> http://williamverdolini.github.io/2014/08/11/cqrses-architecture/
>>> http://docs.geteventstore.com/introduction/3.9.0/event-sourcing-basics/
>>> 
>>> it needs lot of streams to basically replay events for any entity on a
>>> system.
>>> 
>>> example: i could replay events for all changes that happened in 1 Cart of
>>> 1 User:
>>> 
>>> 
>>> (read events from stream "cart-of-user-233293111" ):
>>> 
>>> 1- added item X
>>> 2- deleted item X
>>> 3- added item Y
>>> ....
>>> 
>>> by replaying that stream, I can rebuild a user's cart state
>>> 
>>> 
>>>> Le 28 oct. 2016 à 10:13, Leigh Stewart <lstewart@twitter.com.INVALID>
a
>>> écrit :
>>>> 
>>>> Poule- would you mind sharing some information on Event Sourcing? Are you
>>>> referring to something like
>>>> http://martinfowler.com/eaaDev/EventSourcing.html ?
>>>> 
>>>> On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart <lstewart@twitter.com>
>>> wrote:
>>>> 
>>>>> DL is not able to handle 100s of millions of streams. 10^5-106 is
>>> probably
>>>>> ok.
>>>>> ZK is probably the biggest challenge (we are looking at ways to
>>> eliminate
>>>>> this as we would like to scale to 10^6-10^7 in the not too distant
>>> future),
>>>>> but 100s of millions is so far beyond what we've worked with there would
>>>>> likely be other scaling challenges on the way to that point.
>>>>> 
>>>>> On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue <pouledodue@hotmail.com>
>>>>> wrote:
>>>>> 
>>>>>> In Event Sourcing, we need to have 1 stream per entity/aggregate
so for
>>>>>> a typical prod system it means we need hundreds of millions of streams.
>>>>>> 
>>>>>> Is DL able to handle that or it is limited to, say, few hundreds
>>>>>> thousands of streams?
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>>> 
> 


Mime
View raw message