distributedlog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sijie Guo <si...@apache.org>
Subject Re: hundreds of millions of streams?
Date Tue, 01 Nov 2016 10:17:50 GMT
Poule,

how long do you expect to keep the events for your 1 cart or 1 user? or in
the words, how large is your event stream per cart or per user. it seems
that you want some very lightweight stream/queue solutions - like support
very large amount of streams and per stream throughput is not very high.

- Sijie

On Sun, Oct 30, 2016 at 8:00 AM, Poule Dodue <pouledodue@hotmail.com> wrote:

> i'm not expert but ref. my previous cart example, imho i need a way to
> fetch events of 1 cart or 1 user.
> the actual physical implementation on DL I don't know the technicalities,
> as long I can get 1 cart with the
> less cpu cycles possible.
>
>
> > Le 30 oct. 2016 à 09:59, john.lonergan <john.lonergan@gmail.com> a
> écrit :
> >
> > As an FYI.
> >  It's a misrepresentation to suggest that ES/CQRS requires each
> actor/aggregate root history to be materialised as a physically distinct
> stream. There may be specific implementations that require this, and I know
> there are others that do not.So regarding those references given, it's is
> not a requirement of this tech that we generate one stream in DL per actor
> for ES.It might for instance be necessary for Greg Young's event store to
> have one phys stream per aggregate sure to implementation choices.
> > -------- Original message --------From: Poule Dodue <
> pouledodue@hotmail.com> Date: 28/10/2016  16:24  (GMT+00:00) To:
> dev@distributedlog.incubator.apache.org Subject: Re: hundreds of millions
> of streams?
> > yes aka ES/CQRS
> >
> > some links:
> >
> > https://msdn.microsoft.com/en-us/library/jj554200.aspx
> > http://williamverdolini.github.io/2014/08/11/cqrses-architecture/
> > http://docs.geteventstore.com/introduction/3.9.0/event-sourcing-basics/
> >
> > it needs lot of streams to basically replay events for any entity on a
> system.
> >
> > example: i could replay events for all changes that happened in 1 Cart
> of 1 User:
> >
> >
> > (read events from stream "cart-of-user-233293111" ):
> >
> > 1- added item X
> > 2- deleted item X
> > 3- added item Y
> > ....
> >
> > by replaying that stream, I can rebuild a user's cart state
> >
> >
> >> Le 28 oct. 2016 à 10:13, Leigh Stewart <lstewart@twitter.com.INVALID>
> a écrit :
> >>
> >> Poule- would you mind sharing some information on Event Sourcing? Are
> you
> >> referring to something like
> >> http://martinfowler.com/eaaDev/EventSourcing.html ?
> >>
> >> On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart <lstewart@twitter.com>
> wrote:
> >>
> >>> DL is not able to handle 100s of millions of streams. 10^5-106 is
> probably
> >>> ok.
> >>> ZK is probably the biggest challenge (we are looking at ways to
> eliminate
> >>> this as we would like to scale to 10^6-10^7 in the not too distant
> future),
> >>> but 100s of millions is so far beyond what we've worked with there
> would
> >>> likely be other scaling challenges on the way to that point.
> >>>
> >>> On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue <pouledodue@hotmail.com>
> >>> wrote:
> >>>
> >>>> In Event Sourcing, we need to have 1 stream per entity/aggregate so
> for
> >>>> a typical prod system it means we need hundreds of millions of
> streams.
> >>>>
> >>>> Is DL able to handle that or it is limited to, say, few hundreds
> >>>> thousands of streams?
> >>>>
> >>>>
> >>>>
> >>>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message