distributedlog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "john.lonergan" <john.loner...@gmail.com>
Subject Re: hundreds of millions of streams?
Date Sun, 30 Oct 2016 13:59:31 GMT
As an FYI.
 It's a misrepresentation to suggest that ES/CQRS requires each actor/aggregate root history
to be materialised as a physically distinct stream. There may be specific implementations
that require this, and I know there are others that do not.So regarding those references given,
it's is not a requirement of this tech that we generate one stream in DL per actor for ES.It
might for instance be necessary for Greg Young's event store to have one phys stream per aggregate
sure to implementation choices.
-------- Original message --------From: Poule Dodue <pouledodue@hotmail.com> Date: 28/10/2016
 16:24  (GMT+00:00) To: dev@distributedlog.incubator.apache.org Subject: Re: hundreds of millions
of streams? 
yes aka ES/CQRS

some links:


it needs lot of streams to basically replay events for any entity on a system.

example: i could replay events for all changes that happened in 1 Cart of 1 User:

(read events from stream "cart-of-user-233293111" ):

1- added item X
2- deleted item X
3- added item Y
by replaying that stream, I can rebuild a user's cart state

> Le 28 oct. 2016 à 10:13, Leigh Stewart <lstewart@twitter.com.INVALID> a écrit
> Poule- would you mind sharing some information on Event Sourcing? Are you
> referring to something like
> http://martinfowler.com/eaaDev/EventSourcing.html ?
> On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart <lstewart@twitter.com> wrote:
>> DL is not able to handle 100s of millions of streams. 10^5-106 is probably
>> ok.
>> ZK is probably the biggest challenge (we are looking at ways to eliminate
>> this as we would like to scale to 10^6-10^7 in the not too distant future),
>> but 100s of millions is so far beyond what we've worked with there would
>> likely be other scaling challenges on the way to that point.
>> On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue <pouledodue@hotmail.com>
>> wrote:
>>> In Event Sourcing, we need to have 1 stream per entity/aggregate so for
>>> a typical prod system it means we need hundreds of millions of streams.
>>> Is DL able to handle that or it is limited to, say, few hundreds
>>> thousands of streams?

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message