distributedlog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leigh Stewart <lstew...@twitter.com.INVALID>
Subject Re: hundreds of millions of streams?
Date Fri, 28 Oct 2016 15:48:22 GMT
Got it. We probably can't support that scale at this time.
Curious: do you resort to sharing streams among objects with systems that
don't support 100s millions of streams? (i.e. partitioning objects across
streams?)

On Fri, Oct 28, 2016 at 8:24 AM, Poule Dodue <pouledodue@hotmail.com> wrote:

> yes aka ES/CQRS
>
> some links:
>
> https://msdn.microsoft.com/en-us/library/jj554200.aspx
> http://williamverdolini.github.io/2014/08/11/cqrses-architecture/
> http://docs.geteventstore.com/introduction/3.9.0/event-sourcing-basics/
>
> it needs lot of streams to basically replay events for any entity on a
> system.
>
> example: i could replay events for all changes that happened in 1 Cart of
> 1 User:
>
>
> (read events from stream "cart-of-user-233293111" ):
>
> 1- added item X
> 2- deleted item X
> 3- added item Y
> ....
>
> by replaying that stream, I can rebuild a user's cart state
>
>
> > Le 28 oct. 2016 à 10:13, Leigh Stewart <lstewart@twitter.com.INVALID> a
> écrit :
> >
> > Poule- would you mind sharing some information on Event Sourcing? Are you
> > referring to something like
> > http://martinfowler.com/eaaDev/EventSourcing.html ?
> >
> > On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart <lstewart@twitter.com>
> wrote:
> >
> >> DL is not able to handle 100s of millions of streams. 10^5-106 is
> probably
> >> ok.
> >> ZK is probably the biggest challenge (we are looking at ways to
> eliminate
> >> this as we would like to scale to 10^6-10^7 in the not too distant
> future),
> >> but 100s of millions is so far beyond what we've worked with there would
> >> likely be other scaling challenges on the way to that point.
> >>
> >> On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue <pouledodue@hotmail.com>
> >> wrote:
> >>
> >>> In Event Sourcing, we need to have 1 stream per entity/aggregate so for
> >>> a typical prod system it means we need hundreds of millions of streams.
> >>>
> >>> Is DL able to handle that or it is limited to, say, few hundreds
> >>> thousands of streams?
> >>>
> >>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message