jackrabbit-oak-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Mehrotra <chetan.mehro...@gmail.com>
Subject Oak JCR Observation scalability aspects and concerns
Date Mon, 21 Oct 2013 10:38:43 GMT
Below are the details about discussion held [0] so far on factors
which possibly hinder scaling JCR Observation.
It also touches upon area where usage pattern of applications running
on JCR (like Sling) effects performance

Concerns
-------------

Marcel - Basic concern raised is that listeners without any filter
would cause lots of reads on the repository. these kind of listeners
would pull in modifications of all sessions performing distributed
writes. In our view this will not work well because it puts a very
high load on each of the cluster nodes and will likely delay delivery
of events.

Thomas - As for the theoretical bottleneck, it is quite clear: lets
assume there are 1000 cluster nodes, and each one writes 1000 nodes
per second, there would be 1 million events per second on _each_
cluster node, and 1 billion events per second in the system. It can
not possibly scale. Where exactly the bottleneck is (diffing, creating
the events, whatever) doesn't matter all that much in my view.

Data from an Adobe CQ Instance
------------------------------------------------

To get an idea around kind of JCR observations being used on Oak based
system we pulled in data froma default Adobe CQ setup which uses Oak

Complete details are provided at [1]. Some observations based on the data
* Around 20 listeners are registered for complete repo i.e. at /
* All listeners are using admin user session [2]
* Many listeners are using same set of attributes i.e. filter
  path+user+isDeep. Roughly 24 unique listeners excluding
  Sling JCR installer [3]


Current implementation
----------------------

Current logic works like this for each listener

1. ChangeProcessor gets reported of two states before and after
2. It would pull in tree for two different revisions and under the filter
   path for that listener
3. Then perform a diff which would result in pulling in state at that
   revision from the persistent store. The diff logic is optimized to
   traverse only changed tree paths
4. The diff result would then be filtered based on access check and
   delivered to the listener

So observation logic pull in changes (perform diff) for each listener
separately. Focusing only on one at root we still pull in same set of
information 21 times and then deliver it. This access pattern would
cause such entries to weigh high in the cache (as they are accessed
multiple times) and might occupy more space compared to that actual
importance.

Change the logic such that we have one Oak listener which listens for
changes on root and then deliver to all JCR (after filtering by path
-> nodeType -> user) then we would be able to reduce the number of
calls to persistent store or cache considerably. So it changes current
logic where N listeners independently pull in changes  to one where we
have 1 Oak listener per node which pulls in changes and then delivers
to all. Serving the same role as Sling Listener does for OSGi stack.

Comments
--------

* Marcel - Having large number of fine granied listeners (ones which are
  registered on different path) is better than having listeners registered
  for whole repo

* Thomas - Using a dedicated messaging system (maybe JMS based).As far as
  I understand, some sling / CQ components currently use observation where
  in fact messages wouldn't need to be persisted. At least for some usecases,
  messaging might be an alternative to observation, and would need less
  overhead. I know adding a messaging system would add complexity, but on the
  other hand it might improve performance / scalability because it is
  more flexible than observation.

* Michael Duerig - For layer working above JCR it would help to do some
  analysis of how observation is being used. Something like [4].
  Also when we aggregate listeners we need to ensure that they are
  compatible in terms of filter used, user session etc and it also has to
  take into account listeners lifecycle

* Carsten - We need to confirm where exactly is the performance problem
  Is it when/where the diff is generated?
  When jcr observation events are created based on the diff?
  When the jcr observation events are processed by the listeners?

  Also we can possibly modify the Sling eventing such that
  a) it uses an Oak API (instead of JCR Observation) for increased
     efficiency and
  b) adds more data to the the OSGi events, so that subsequent consumers
     can readily access the data without additional reads.

Next Steps
----------

* Look into the possibility to change design aggregates listeners and thus
  reduce number of call to backened persistent system
* Optimize the default setup for fewer number of cluster nodes
* Provide a way to scale for large number of cluster nodes i.e. say using
  external messaging system
* Collect more data around current design to confirm is this a problem at all :)

If I missed on any part kindly add it to the discussion.

regards
Chetan
[0] Participants - Michael Marth, Thomas Mueller, Marcel Reutegger,
    Michael Duerig, Carsten Ziegeler, Chetan Mehrotra
[1] https://gist.github.com/chetanmeh/7081328
[2] https://gist.github.com/chetanmeh/7081328/raw/listeners-list-filtered.txt
[3] https://git.corp.adobe.com/gist/chetanm/863/raw/listerners-per-path.txt
[4] https://cwiki.apache.org/confluence/display/SLING/Observation+usage+patterns

Mime
View raw message