jackrabbit-oak-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lukas Eder <lukas.e...@gmail.com>
Subject Re: Scalability of JCR observation
Date Wed, 17 Apr 2013 07:34:03 GMT
2013/4/17 Jukka Zitting <jukka.zitting@gmail.com>:
> Hi,
> On Wed, Apr 17, 2013 at 3:35 AM, Ian Boston <ieb@tfd.co.uk> wrote:
>> When mentioning AMQ, I forgot to mention that one size doesn't fit all use
>> cases. When I first connected AMQ to Jackrabbit observation via OSGi
>> events, AMQ rapidly became swamped with the volume of events. OSGi events
>> were just about able to cope but as soon as the events were propagated over
>> the network the volume was too great.
> Indeed. Most message queue systems out there are designed for
> application/document level messages (like "document added", "action
> taken", etc.), i.e. the granularity of events is generally coarser
> than the field-level events ("property changed", "node added", etc.)
> you get with JCR observation.
> My thinking here is that the best approach for most use cases will be
> to use a commit hook to consolidate the low level changes to less
> granular application events that can then be processed in the
> background using JCR observation, OSGi events, a generic message queue
> system, or directly by a polling thread in the client application.

I agree with this line of thinking.

Instead of doing any performance improvement or event aggregation at a
low-level event producer site, I think that Oak itself should generate
as many meaningful events as possible. With a sensible eventing SPI,
it is easy for an Oak user to implement "event listener pipes" or
"event transformers" which filter, aggregate, and package low-level
events into fewer but bigger messages, depending on the concrete event
distribution requirements. The originally suggested aggregation on a
revision level would be an example for such a "custom" aggregation.

If MQ or OSGi event listener implementations are flooded too quickly,
Oak (or an Oak extension module) could provide a sample implementation
for event accumulation (at the producer site) and dispersion (at the
consumer site), to transparently package events into fewer but bigger
physical messages for transport - while still being able to consume
each low-level event at the consumer site if needed.

Oak itself, however, shouldn't make too many architecture / topology
assumptions about the components that build on it. I feel that
concrete implementation facts (internal and external to Oak) are
beginning to leak into Oak's eventing SPI.


View raw message