incubator-sling-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Sling Website > Apache Sling Eventing and Job Handling
Date Mon, 18 Oct 2010 14:19:00 GMT
<html>
<head>
    <base href="https://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/1810/9/1/_/styles/combined.css?spaceKey=SLINGxSITE&amp;forWysiwyg=true" type="text/css">
    </head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
    <h2><a href="https://cwiki.apache.org/confluence/display/SLINGxSITE/Apache+Sling+Eventing+and+Job+Handling">Apache Sling Eventing and Job Handling</a></h2>
    <h4>Page <b>edited</b> by             <a href="https://cwiki.apache.org/confluence/display/~cziegeler@apache.org">Carsten Ziegeler</a>
    </h4>
        <br/>
                         <h4>Changes (2)</h4>
                                 
    
<div id="page-diffs">
            <table class="diff" cellpadding="0" cellspacing="0">
            <tr><td class="diff-added-lines" style="background-color: #dfd;">h2. Overview <br></td></tr>
            <tr><td class="diff-unchanged" > <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">The Apache Sling Event Support bundle provides interesting services for advanced event handling and job processing. While this bundle leverages the OSGi EventAdmin, it provides a very powerful support for so called jobs: a job is a task which has to be performed by a component - the Sling job handling ensures that exactly one component performs this task. <br> <br>To get some hands on code, you can refer to the following tutorials: <br>* [How to Manage Events in Sling] <br>* [Scheduler Service (commons scheduler)] <br> <br>h3. Possible Use Cases for Job Handling <br>* Workflow <br>* Post Processing (business processes) <br>* Caching <br>* Auditing <br> <br>h2. Distributed Events <br> <br>In addition to the job handling, the Sling Event support adds handling for distributed events. A distributed event is an OSGi event which is sent across JVM boundaries to a different VM. A potential use case is to broadcast information in a clustered environment. <br> <br>h3. Sources of Events <br> <br>When it comes to application based on Sling, there is a variety of sources from which OSGi events can be send: <br>* JCR observation events <br>* Application generated events <br>* Events from messaging systems (~JMS) <br>* &quot;External events&quot; <br> <br>The events can eiter be generated inside a current user context, e.g. when the user performs an action through the UI, or they can be out of a user context, e.g. for schedulded events. This leads to different weights of events. <br> <br>h3. Weights of Events <br> <br>We can distinguish two different weights of events, depending how they are distributed in a clustered environment: <br> <br> * User generated events - these events are generated directly by some user action and are therefore started on one single node. <br> * Environment generated events (JCR events, scheduler events etc.) - these events are generated &quot;simultanously&quot; on all nodes. <br> <br>External events, like incoming JMS events etc. might fall either into the first or the second category. The receiver of such events must have the knowledge about the weight of the event. <br> <br>h3. Basic Principles <br> <br>The foundation of the distributed event mechanism is to distribute each event to every node in a clustered environment. The event distribution mechanism has no knowledge about the intent of the event and therefore is not able to make delivery decisions by itself. It is up to the sender to decide what should happen, however the sender must explicitly declare an event to be distributed. There are exceptions to &quot;distributing everything to everywhere&quot; as for example framework related events (bundle stopped, installed etc.) should not be distributed. <br> <br>The event mechanism will provide additional functionality making it easier for event receivers to decide if they should process an event. The event receiver can determine if the event is a local event or comming from a remote application node. Therefore a general rule of thumb is to process events only if they&#39;re local and just regard remote events as a FYI. <br> <br>The event mechanism is an *event* mechanism which should not be confused with a *messaging* mechanism. Events are received by the event mechanism and distributed to registered listeners. Concepts like durable listeners, guarantee of processing etc. are not part of the event mechanism itself. However, there is additional support for such things, like job handling. <br> <br>The application should try to use application events instead of low level JCR events whereever possible. Therefore a bridging between JCR events and the event mechanism is required. However, a general &quot;automatic&quot; mapping will not be provided. It is up to the application to develop such a mapping on a per use case base. There might be some support to make the mapping easier. <br> <br>The event handling should be made as transparent to the developer as possible. Therefore the additional code for a developer to make the eventing working in a clustered environment etc. should be kept to a minimum (which will hopefully reduce possible user errors). <br> <br>h2. Event Mechanism <br> <br>The event mechanism is leveraging the OSGi Event Admin Specification (OSGi Compendium 113). The event admin specification provides a sufficient base. It is based on the event publish and subscribe mechanism. Each event is associated with a topic and data. The data consists of custom key-value pairs where the keys are strings and the values can be any object. However, to work in distributed environments it is advisable to use only string and scalar types for data. If complex objects are required they have to be serializable. <br> <br>Events can either be send synchronously or asynchronously. It is up to the caller (the one sending the event) to decide this by choosing one of the provided methods. <br> <br>The OSGi API is very simple and leightweight - sending an event is just generating the event object and calling the event admin. Rceiving the event is implementing a single interface and declaring through properties which topics one is interested in. It&#39;s possible to add an additional filter (based on property values for example). <br> <br>%N The event handler should not take too much time to process the event. For example, the Apache Felix implementation of the event admin black lists an event handler if it takes more than 5 seconds to process the event - regardless if the event is sent synchronously or asynchronously. Therefore any heavier processing has to be done in the background. The event is just the trigger to start this. The job mechanism explained in this documentation is a good way of implementing this functionality for an event handler. <br> <br>The aim is to add all functionality on top of an existing event admin implementation. Therefore everything should be added by additional event handlers. <br> <br>h2. Event Handler <br> <br>An event handler registers itself on a (set of) topic. It can also specify a filter for the events. This event handler is either notified synchronously or asynchronously depending on how the event has been sent. <br> <br>h2. Events <br> <br>The type of the event is specified by the hierarchically organized topic. In order to provide clustering of JCR repositories and clustering of the sling based application instances, each event can contain the following properties - if they are absent, a default value is assumed: <br> <br>* =event.distribute= - this flag is set by the sender of an event to give a hint if the event should be distributed across instances. For example JCR observation based events are already distributed on all instances, so there is no further need to distribute them. If the flag is present, the event will be distributed. The value has currently no meaning, however the EventUtil method should be used to add this property. If the flag is absent the event is distributed locally only. <br>* =event.application= - An identifier for the current application node in the cluster. This information will be used to detect if an event has been created on different nodes. If the event has been created on the same note, the =event.application= is missing, if it is a remote event, the =event.application= contains the ID of the node, the event has been initially created. <br> <br>While the =event.distribute= must be set by the sender of an event (if the event should be distributed), the =event.application= property is maintained by the event mechanism. Therefore a client sending an event should *never* set this information by itself. This will confuse the local event handlers and result in unexpected behaviour. On remote events the =event.application= is set by the event distribution mechanism. <br> <br>h2. Event Distribution Across Application Nodes (Cluster) <br> <br>The (local) event admin is the service distributing events locally. The Sling Distributing Event Handler is a registered event handler that is listening for events to be distributed. It distributes the events to remote application notes, the JCR repository is used for distribution. The distributing event handler writes the events into the repository, the distributing event handlers on other application nodes get notified through observation and then distribute the read events locally. <br> <br>As mentioned above, the client sending an event has to mark an event to be distributed in a cluster by setting the =event.distribute= in the event properties (through ~EventUtil). <br>The existance of this flag allows to register an event handler for all events having this flag. The event handler will add the =event.application= information and write the event into the repository. All other application nodes have an observer registered and get notified each time a new event is added to the repository. They&#39;ll read the event from the repository, clear the =event.distribute= and send this event locally and asynchronously. <br> <br>An event handler receiving such an event can distinguish it by checking the =event.application= property. If the property is not available, it is a local event - if the property is available it is a remote event. <br> <br>This distribution mechanism has the advantage that the application nodes do not need to know each other and the distribution mechanism is independent from the used event admin implementation. Defining the filter for the =event.distribute= is also very simple. <br> <br>h3. Storing Events in the Repository <br> <br>Distributable events are stored in the repository, the repository will have a specific area (path) where all events are stored.  <br> <br>Each event is stored as a separate node with the following properties: <br>| *Property Name*     | *Description* | <br>| =event:topic=       | The topic of the event | <br>| =event:application= | The identifier of the application node where the event was created | <br>| =event:created=     | The date and time when the event has been created (stored in the repository) <br>| =event:properties=  | Serialized properties (except the =event.distribute=, but including the =event.application=) | <br> <br>Each application is periodically removing old events from the repository (using the scheduler). <br> <br>h2. Jobs (Guarantee of Processing) <br> <br>In general, the eventing mechanism has no knowledge about the contents of an event. Therefore, it can&#39;t decide if an event must be processed by a node. As the event mechanism is a &quot;fire event and forget about it&quot; algorithm, there is no way for an event admin to tell if someone has processed the event. <br> <br>On the other hand, there are use cases where the guarantee of processing is a must and usually this comes with the requirement of processing this event exactly once. Typical examples are sending notification emails (or sms) or post processing of content (like thumbnail generation of images or documents). <br> <br>We will call these events jobs to make clear that someone has to do something with the event (do the job). We will use a special topic =org/apache/sling/event/job= to indicate that the event contains a job, the real topic of the event is stored in the =event.job.topic= property. When a job event (event with the topic =org/apache/sling/event/job=) is received, a new event with the topic from the property =event.job.topic= is fired. <br> <br>The event must have the following properties: <br>| *Property Name*   | *Description* | <br>| =event.job.topic= | The topic of the job | <br>| =event.job.id=    | A unique identifier for this job (optional) | <br> <br>The job event handler listens for all job events (all events with the topic =org/apache/sling/event/job=). The event handler will write the job event into the repository (into the job area), lock it, create a new event with the topic from the property =event.job.topic= and send the job event through the event admin. When the job is finished, the event listener will unlock the node from the repository. <br> <br>To avoid timeouts and black listing of event handlers, the job event handler does not assume that the job has been processed if the event could be sent successfully. It is the task of the event handler to notify the job event handler that it has processed the job. In addition, the job processing should be done in the background. The =EventUtil= class has a helper method for this: =processJob(Event, JobProcessor)=. The event handler must implement the =JobProcessor= interface which consists of a single =process(Event)= method. When the event handler receives a job event, it calls =EventUtil.processJob(event, this)= and returns. The =process(Event)= method is now called in the background and when it finishes, the job event handler is notified that the job is completed. <br> <br>If the event handler wants to do the background processing by itself or does not need background processing at all, it must signal completition of the job by call =EventUtil.finishedJob(event)=. <br> <br>By default an application node is queuing the jobs which means that only one job is processed at a time. If a job can be run in parallel on one application node, the property =event.job.parallel= should be set with any value. <br> <br>The job id is optional and can be used to update or reactivate jobs. <br> <br>h3. Storing Jobs in the Repository <br> <br>Jobs are stored in the repository in order to ensure that exactly one single application node is processing the job. The repository will have a specific area (path) where all jobs are stored. In order to distinguish a job which occured twice and a job which is generated at the same time on several nodes, each job can be uniquely identified by its topic (property =event.job.topic=) and the =event.job.id= property. It is up to the client who is creating the event to ensure that the =event.job.id= property is unqiue *and* identical on all application nodes. If the job id is not provided for the job, then it is up to the client to ensure that the job even is only fired once. <br> <br>When the job event listener tries to write a job into the repository it will check if the repository already contains a job with the given topic =event.job.topic= and =event.job.id= property. If the event has already been written by some other application node, it&#39;s not written again. If the event has been written by the same node, it will be set to active again (=event:active= will be set to true and =event:created= will be updated). <br> <br>Each job is stored as a separate node with the following properties: <br>| *Property Name*     | *Description* | <br>| =event:topic=       | The topic of the job | <br>| =event:application= | The identifier of the node where the job was created | <br>| =event:processor=   | The identifier of the node which processed the job | <br>| =event:active=      | Indicates if this job is active and should be processed(unlocked) or is currently processed (locked) | <br>| =event:created=     | The date and time when the event has been created (stored in the repository) <br>| =event:id=          | The unique identifier of this job (optional). <br>| =event:properties=  | Serialized properties | <br>| =event:finished=    | The date and time when the job has been finished | <br> <br> <br>The failover of an application node is accomplished by locking and the =event:active= flag. If a job is locked in the repository a session scoped lock is used. If this application node dies, the lock dies as well. Each application node observes the JCR locking properties and therefore gets aware of unlocked event nodes with the active flag set to true. If an application node finds such a node, it locks it, updates the =event:application= information and processes it accordingly. In this case the event gets the additional property =org/apache/sling/job/retry=.  <br> <br>Each application is periodically removing old jobs from the repository (using the scheduler). <br> <br>h3. Distribution of Jobs <br> <br>A job event is an event like any other. Therefore it is up to the client generating the event to decide if the event should be distributed. If the event is distributed, it will be distributed with a set =event.application= on the remote nodes. If the job event handler receives a job with the =event.application= property set, it will not try to write it into the repository. It will just broadcast this event asynchronously as a ~FYI event. <br> <br>If a job event is created simultanously on all application nodes, the event will not be distributed. The application node that actually has the lock on the stored job in the repository will clear the =event.application= when sending the event locally. All other application nodes will use the =event.application= stored in the repository when broadcasting the event locally. <br> <br>h2. Usage Patterns <br> <br>Based on some usage patterns, we discuss the functionality of the eventing mechanism. <br> <br>h3. Sending User Generated Events <br> <br>If a user action results in an event, the event is only created on one single node in the cluster. The event object is generated and delivered to the OSGi event admin. If the =event.distribute= is not explicitly set, the event is only distributed localled. <br> <br>If the =event.distribute= is the, the cluster event handler will write the event into the repository. All nodes in the cluster observe the repository area where all events are stored. If a new event is written into that area, each application node will get notified. It will create the event based on the information in the repository, clear the =event.distribute= and publish the event. <br> <br>The flow can be described as follows: <br># Client code generates event using OSGi API, if the =event.distribute= should be set, it is using the ~EventUtil. <br># Client code sends the event to the (local) event admin. <br># Event admin delivers the event locally. <br># Clustering event handler receives the event if =event.distribute= is present <br>## Event handler adds =event.application= and writes the event to the repository <br>## Remote repository observers get notified through JCR observation about the new event. They distribute the event locally with the =event.application= (from the node where the event occured first) and cleared =event.distribute=. <br> <br>h3. Processing JCR Events <br> <br>JCR events are environment generated events and therefore are sent by the repository to each node in the cluster. In general, it is advisable to not built the application on the low level repository events but to use application events. Therefore the observer of the JCR event should create an OSGi event based on the changes in the repository. A decision has to be made if the event should be a job or a plain event. <br> <br>The flow can be described as follows: <br># Client registers for JCR observation <br># JCR notifies the client for changes <br># Client generates OSGi event based on the JCR events (the =event.distribute= will not be set), it decides if it sends this event as a job. <br># Client code sends the event to the (local) event admin <br># Event admin publishes the event locally <br># The distribution event handler does not set see the event as the =event.distribute= is not set. <br># The job event handler gets the event if it has the job topic <br>## The job event handler adds the =event.application= property and tries to write the job to the repository <br>### If no job with the topic and =id= property is in the repository, the event will be written and locked. <br>### If an event with the topic and =id= property is in the repository then: <br>#### If the =event.application= equals the current application node, the event is set to active (=event:active=) in the repository again and locked <br>#### If the =event.application= does not equal the current application node, the event is not distributed locally. <br>### If the job could be locked in the repository, the job event handler delivers the job locally and synchronously and it unlocks the job and sets =event:active= to false afterwards. <br> <br>h3. Sending Scheduled Events <br> <br>Scheduled events are OSGi events that have been created by the environemnt. They are generated on each application node of the cluster through an own scheduler instance. Sending these events works the same as sending events based on JCR events (see above). <br> <br>In most use cases a scheduler will send job events to ensure that exactly one application node is processing the event. <br> <br>h3. Receiving OSGi Events <br> <br>If you want to receive OSGi events, you can just follow the specification: receive it via a custom event handler which is registered on bundle start - a filter can be specified as a configuration property of the handler.  <br> <br>As we follow the principle of distributing each event to every registered handler, the handler has to decide if it will process the event. In order to avoid multiple processing of this event in a clustered environment, the event handler should check the =event.application= property. If it is not set, it&#39;s a local event and the handler should process the event. If the =event.application= is set, it&#39;s a remote event and the handler should not process the event. This is a general rule of thumb - however, it&#39;s up to the handler to make its decision either on =event.application= or any other information. <br> <br>It is advisable to perform the local event check even in a non clustered environment as it makes the migration to a cluster later on much easier and there is nearly no performance overhead caused by the check. <br> <br>The ~EventUtil class provides an utility method =isLocalEvent(Event)= which checks the existance of the =event.application= property and returns =true= if it is absend. <br> <br>h2. Scheduler <br> <br>Each Sling based application will contain a scheduler service (which is based on the Quartz open source project). <br> <br>h2. Use Cases <br> <br>h3. Post Processing (Business Processes) <br> <br>A typical example for post processing (or running a business process) is sending an email or creating thumbnails and extracting meta data from the content (like we do in DAM), which we will discuss here. <br> <br>An appropriate JCR observer will be registered. This observer detects when new content is put into the repository or when content is changed. In these cases it creates appropriate =CONTENT_ADDED=, =CONTENT_UPDATED= OSGi events from the JCR events. In order to ensure that these actions get processed accordingly, the event is send as a job (with the special job topic, the =topic= and =id= property). <br> <br>The event admin now delivers these jobs to the registered handlers. The job event handler gets notified and (simplified version) sends the contained event synchronously. One of the handlers for these events is the post processing service in DAM. The job mechanism ensures that exactly one application node is post processing and that the process has to be finished even if the application node dies during execution. <br> <br>h2. Scheduling <br> <br>The scheduler is a service which uses the open source Quartz library. The scheduler has methods to start jobs periodically or with a cron definition. In addition, a service either implementing =java.lang.Runnable= or =org.quartz.job= is started through the whiteboard pattern *if* it either contains a configuration property =scheduler.expression= or =scheduler.period=. The job is started with the ~PID of the service - if the service has no PID, the configuration property =scheduler.name= must be set. <br></td></tr>
        </table>
</div>                            <h4>Full Content</h4>
                    <div class="notificationGreySide">
        <h2><a name="ApacheSlingEventingandJobHandling-Overview"></a>Overview</h2>

<p>The Apache Sling Event Support bundle provides interesting services for advanced event handling and job processing. While this bundle leverages the OSGi EventAdmin, it provides a very powerful support for so called jobs: a job is a task which has to be performed by a component - the Sling job handling ensures that exactly one component performs this task.</p>

<p>To get some hands on code, you can refer to the following tutorials:</p>
<ul>
	<li><a href="/confluence/display/SLINGxSITE/How+to+Manage+Events+in+Sling" title="How to Manage Events in Sling">How to Manage Events in Sling</a></li>
	<li><a href="/confluence/display/SLINGxSITE/Scheduler+Service+%28commons+scheduler%29" title="Scheduler Service (commons scheduler)">Scheduler Service &#40;commons scheduler&#41;</a></li>
</ul>


<h3><a name="ApacheSlingEventingandJobHandling-PossibleUseCasesforJobHandling"></a>Possible Use Cases for Job Handling</h3>
<ul>
	<li>Workflow</li>
	<li>Post Processing (business processes)</li>
	<li>Caching</li>
	<li>Auditing</li>
</ul>


<h2><a name="ApacheSlingEventingandJobHandling-DistributedEvents"></a>Distributed Events</h2>

<p>In addition to the job handling, the Sling Event support adds handling for distributed events. A distributed event is an OSGi event which is sent across JVM boundaries to a different VM. A potential use case is to broadcast information in a clustered environment.</p>

<h3><a name="ApacheSlingEventingandJobHandling-SourcesofEvents"></a>Sources of Events</h3>

<p>When it comes to application based on Sling, there is a variety of sources from which OSGi events can be send:</p>
<ul>
	<li>JCR observation events</li>
	<li>Application generated events</li>
	<li>Events from messaging systems (~JMS)</li>
	<li>"External events"</li>
</ul>


<p>The events can eiter be generated inside a current user context, e.g. when the user performs an action through the UI, or they can be out of a user context, e.g. for schedulded events. This leads to different weights of events.</p>

<h3><a name="ApacheSlingEventingandJobHandling-WeightsofEvents"></a>Weights of Events</h3>

<p>We can distinguish two different weights of events, depending how they are distributed in a clustered environment:</p>

<ul>
	<li>User generated events - these events are generated directly by some user action and are therefore started on one single node.</li>
	<li>Environment generated events (JCR events, scheduler events etc.) - these events are generated "simultanously" on all nodes.</li>
</ul>


<p>External events, like incoming JMS events etc. might fall either into the first or the second category. The receiver of such events must have the knowledge about the weight of the event.</p>

<h3><a name="ApacheSlingEventingandJobHandling-BasicPrinciples"></a>Basic Principles</h3>

<p>The foundation of the distributed event mechanism is to distribute each event to every node in a clustered environment. The event distribution mechanism has no knowledge about the intent of the event and therefore is not able to make delivery decisions by itself. It is up to the sender to decide what should happen, however the sender must explicitly declare an event to be distributed. There are exceptions to "distributing everything to everywhere" as for example framework related events (bundle stopped, installed etc.) should not be distributed.</p>

<p>The event mechanism will provide additional functionality making it easier for event receivers to decide if they should process an event. The event receiver can determine if the event is a local event or comming from a remote application node. Therefore a general rule of thumb is to process events only if they're local and just regard remote events as a FYI.</p>

<p>The event mechanism is an <b>event</b> mechanism which should not be confused with a <b>messaging</b> mechanism. Events are received by the event mechanism and distributed to registered listeners. Concepts like durable listeners, guarantee of processing etc. are not part of the event mechanism itself. However, there is additional support for such things, like job handling.</p>

<p>The application should try to use application events instead of low level JCR events whereever possible. Therefore a bridging between JCR events and the event mechanism is required. However, a general "automatic" mapping will not be provided. It is up to the application to develop such a mapping on a per use case base. There might be some support to make the mapping easier.</p>

<p>The event handling should be made as transparent to the developer as possible. Therefore the additional code for a developer to make the eventing working in a clustered environment etc. should be kept to a minimum (which will hopefully reduce possible user errors).</p>

<h2><a name="ApacheSlingEventingandJobHandling-EventMechanism"></a>Event Mechanism</h2>

<p>The event mechanism is leveraging the OSGi Event Admin Specification (OSGi Compendium 113). The event admin specification provides a sufficient base. It is based on the event publish and subscribe mechanism. Each event is associated with a topic and data. The data consists of custom key-value pairs where the keys are strings and the values can be any object. However, to work in distributed environments it is advisable to use only string and scalar types for data. If complex objects are required they have to be serializable.</p>

<p>Events can either be send synchronously or asynchronously. It is up to the caller (the one sending the event) to decide this by choosing one of the provided methods.</p>

<p>The OSGi API is very simple and leightweight - sending an event is just generating the event object and calling the event admin. Rceiving the event is implementing a single interface and declaring through properties which topics one is interested in. It's possible to add an additional filter (based on property values for example).</p>

<p>%N The event handler should not take too much time to process the event. For example, the Apache Felix implementation of the event admin black lists an event handler if it takes more than 5 seconds to process the event - regardless if the event is sent synchronously or asynchronously. Therefore any heavier processing has to be done in the background. The event is just the trigger to start this. The job mechanism explained in this documentation is a good way of implementing this functionality for an event handler.</p>

<p>The aim is to add all functionality on top of an existing event admin implementation. Therefore everything should be added by additional event handlers.</p>

<h2><a name="ApacheSlingEventingandJobHandling-EventHandler"></a>Event Handler</h2>

<p>An event handler registers itself on a (set of) topic. It can also specify a filter for the events. This event handler is either notified synchronously or asynchronously depending on how the event has been sent.</p>

<h2><a name="ApacheSlingEventingandJobHandling-Events"></a>Events</h2>

<p>The type of the event is specified by the hierarchically organized topic. In order to provide clustering of JCR repositories and clustering of the sling based application instances, each event can contain the following properties - if they are absent, a default value is assumed:</p>

<ul>
	<li>=event.distribute= - this flag is set by the sender of an event to give a hint if the event should be distributed across instances. For example JCR observation based events are already distributed on all instances, so there is no further need to distribute them. If the flag is present, the event will be distributed. The value has currently no meaning, however the EventUtil method should be used to add this property. If the flag is absent the event is distributed locally only.</li>
	<li>=event.application= - An identifier for the current application node in the cluster. This information will be used to detect if an event has been created on different nodes. If the event has been created on the same note, the =event.application= is missing, if it is a remote event, the =event.application= contains the ID of the node, the event has been initially created.</li>
</ul>


<p>While the =event.distribute= must be set by the sender of an event (if the event should be distributed), the =event.application= property is maintained by the event mechanism. Therefore a client sending an event should <b>never</b> set this information by itself. This will confuse the local event handlers and result in unexpected behaviour. On remote events the =event.application= is set by the event distribution mechanism.</p>

<h2><a name="ApacheSlingEventingandJobHandling-EventDistributionAcrossApplicationNodes%28Cluster%29"></a>Event Distribution Across Application Nodes (Cluster)</h2>

<p>The (local) event admin is the service distributing events locally. The Sling Distributing Event Handler is a registered event handler that is listening for events to be distributed. It distributes the events to remote application notes, the JCR repository is used for distribution. The distributing event handler writes the events into the repository, the distributing event handlers on other application nodes get notified through observation and then distribute the read events locally.</p>

<p>As mentioned above, the client sending an event has to mark an event to be distributed in a cluster by setting the =event.distribute= in the event properties (through ~EventUtil).<br/>
The existance of this flag allows to register an event handler for all events having this flag. The event handler will add the =event.application= information and write the event into the repository. All other application nodes have an observer registered and get notified each time a new event is added to the repository. They'll read the event from the repository, clear the =event.distribute= and send this event locally and asynchronously.</p>

<p>An event handler receiving such an event can distinguish it by checking the =event.application= property. If the property is not available, it is a local event - if the property is available it is a remote event.</p>

<p>This distribution mechanism has the advantage that the application nodes do not need to know each other and the distribution mechanism is independent from the used event admin implementation. Defining the filter for the =event.distribute= is also very simple.</p>

<h3><a name="ApacheSlingEventingandJobHandling-StoringEventsintheRepository"></a>Storing Events in the Repository</h3>

<p>Distributable events are stored in the repository, the repository will have a specific area (path) where all events are stored. </p>

<p>Each event is stored as a separate node with the following properties:</p>
<div class='table-wrap'>
<table class='confluenceTable'><tbody>
<tr>
<td class='confluenceTd'> <b>Property Name</b>     </td>
<td class='confluenceTd'> <b>Description</b> </td>
</tr>
<tr>
<td class='confluenceTd'> =event:topic=       </td>
<td class='confluenceTd'> The topic of the event </td>
</tr>
<tr>
<td class='confluenceTd'> =event:application= </td>
<td class='confluenceTd'> The identifier of the application node where the event was created </td>
</tr>
<tr>
<td class='confluenceTd'> =event:created=     </td>
<td class='confluenceTd'> The date and time when the event has been created (stored in the repository)</td>
</tr>
<tr>
<td class='confluenceTd'> =event:properties=  </td>
<td class='confluenceTd'> Serialized properties (except the =event.distribute=, but including the =event.application=) </td>
</tr>
</tbody></table>
</div>


<p>Each application is periodically removing old events from the repository (using the scheduler).</p>

<h2><a name="ApacheSlingEventingandJobHandling-Jobs%28GuaranteeofProcessing%29"></a>Jobs (Guarantee of Processing)</h2>

<p>In general, the eventing mechanism has no knowledge about the contents of an event. Therefore, it can't decide if an event must be processed by a node. As the event mechanism is a "fire event and forget about it" algorithm, there is no way for an event admin to tell if someone has processed the event.</p>

<p>On the other hand, there are use cases where the guarantee of processing is a must and usually this comes with the requirement of processing this event exactly once. Typical examples are sending notification emails (or sms) or post processing of content (like thumbnail generation of images or documents).</p>

<p>We will call these events jobs to make clear that someone has to do something with the event (do the job). We will use a special topic =org/apache/sling/event/job= to indicate that the event contains a job, the real topic of the event is stored in the =event.job.topic= property. When a job event (event with the topic =org/apache/sling/event/job=) is received, a new event with the topic from the property =event.job.topic= is fired.</p>

<p>The event must have the following properties:</p>
<div class='table-wrap'>
<table class='confluenceTable'><tbody>
<tr>
<td class='confluenceTd'> <b>Property Name</b>   </td>
<td class='confluenceTd'> <b>Description</b> </td>
</tr>
<tr>
<td class='confluenceTd'> =event.job.topic= </td>
<td class='confluenceTd'> The topic of the job </td>
</tr>
<tr>
<td class='confluenceTd'> =event.job.id=    </td>
<td class='confluenceTd'> A unique identifier for this job (optional) </td>
</tr>
</tbody></table>
</div>


<p>The job event handler listens for all job events (all events with the topic =org/apache/sling/event/job=). The event handler will write the job event into the repository (into the job area), lock it, create a new event with the topic from the property =event.job.topic= and send the job event through the event admin. When the job is finished, the event listener will unlock the node from the repository.</p>

<p>To avoid timeouts and black listing of event handlers, the job event handler does not assume that the job has been processed if the event could be sent successfully. It is the task of the event handler to notify the job event handler that it has processed the job. In addition, the job processing should be done in the background. The =EventUtil= class has a helper method for this: =processJob(Event, JobProcessor)=. The event handler must implement the =JobProcessor= interface which consists of a single =process(Event)= method. When the event handler receives a job event, it calls =EventUtil.processJob(event, this)= and returns. The =process(Event)= method is now called in the background and when it finishes, the job event handler is notified that the job is completed.</p>

<p>If the event handler wants to do the background processing by itself or does not need background processing at all, it must signal completition of the job by call =EventUtil.finishedJob(event)=.</p>

<p>By default an application node is queuing the jobs which means that only one job is processed at a time. If a job can be run in parallel on one application node, the property =event.job.parallel= should be set with any value.</p>

<p>The job id is optional and can be used to update or reactivate jobs.</p>

<h3><a name="ApacheSlingEventingandJobHandling-StoringJobsintheRepository"></a>Storing Jobs in the Repository</h3>

<p>Jobs are stored in the repository in order to ensure that exactly one single application node is processing the job. The repository will have a specific area (path) where all jobs are stored. In order to distinguish a job which occured twice and a job which is generated at the same time on several nodes, each job can be uniquely identified by its topic (property =event.job.topic=) and the =event.job.id= property. It is up to the client who is creating the event to ensure that the =event.job.id= property is unqiue <b>and</b> identical on all application nodes. If the job id is not provided for the job, then it is up to the client to ensure that the job even is only fired once.</p>

<p>When the job event listener tries to write a job into the repository it will check if the repository already contains a job with the given topic =event.job.topic= and =event.job.id= property. If the event has already been written by some other application node, it's not written again. If the event has been written by the same node, it will be set to active again (=event:active= will be set to true and =event:created= will be updated).</p>

<p>Each job is stored as a separate node with the following properties:</p>
<div class='table-wrap'>
<table class='confluenceTable'><tbody>
<tr>
<td class='confluenceTd'> <b>Property Name</b>     </td>
<td class='confluenceTd'> <b>Description</b> </td>
</tr>
<tr>
<td class='confluenceTd'> =event:topic=       </td>
<td class='confluenceTd'> The topic of the job </td>
</tr>
<tr>
<td class='confluenceTd'> =event:application= </td>
<td class='confluenceTd'> The identifier of the node where the job was created </td>
</tr>
<tr>
<td class='confluenceTd'> =event:processor=   </td>
<td class='confluenceTd'> The identifier of the node which processed the job </td>
</tr>
<tr>
<td class='confluenceTd'> =event:active=      </td>
<td class='confluenceTd'> Indicates if this job is active and should be processed(unlocked) or is currently processed (locked) </td>
</tr>
<tr>
<td class='confluenceTd'> =event:created=     </td>
<td class='confluenceTd'> The date and time when the event has been created (stored in the repository)</td>
</tr>
<tr>
<td class='confluenceTd'> =event:id=          </td>
<td class='confluenceTd'> The unique identifier of this job (optional).</td>
</tr>
<tr>
<td class='confluenceTd'> =event:properties=  </td>
<td class='confluenceTd'> Serialized properties </td>
</tr>
<tr>
<td class='confluenceTd'> =event:finished=    </td>
<td class='confluenceTd'> The date and time when the job has been finished </td>
</tr>
</tbody></table>
</div>



<p>The failover of an application node is accomplished by locking and the =event:active= flag. If a job is locked in the repository a session scoped lock is used. If this application node dies, the lock dies as well. Each application node observes the JCR locking properties and therefore gets aware of unlocked event nodes with the active flag set to true. If an application node finds such a node, it locks it, updates the =event:application= information and processes it accordingly. In this case the event gets the additional property =org/apache/sling/job/retry=. </p>

<p>Each application is periodically removing old jobs from the repository (using the scheduler).</p>

<h3><a name="ApacheSlingEventingandJobHandling-DistributionofJobs"></a>Distribution of Jobs</h3>

<p>A job event is an event like any other. Therefore it is up to the client generating the event to decide if the event should be distributed. If the event is distributed, it will be distributed with a set =event.application= on the remote nodes. If the job event handler receives a job with the =event.application= property set, it will not try to write it into the repository. It will just broadcast this event asynchronously as a ~FYI event.</p>

<p>If a job event is created simultanously on all application nodes, the event will not be distributed. The application node that actually has the lock on the stored job in the repository will clear the =event.application= when sending the event locally. All other application nodes will use the =event.application= stored in the repository when broadcasting the event locally.</p>

<h2><a name="ApacheSlingEventingandJobHandling-UsagePatterns"></a>Usage Patterns</h2>

<p>Based on some usage patterns, we discuss the functionality of the eventing mechanism.</p>

<h3><a name="ApacheSlingEventingandJobHandling-SendingUserGeneratedEvents"></a>Sending User Generated Events</h3>

<p>If a user action results in an event, the event is only created on one single node in the cluster. The event object is generated and delivered to the OSGi event admin. If the =event.distribute= is not explicitly set, the event is only distributed localled.</p>

<p>If the =event.distribute= is the, the cluster event handler will write the event into the repository. All nodes in the cluster observe the repository area where all events are stored. If a new event is written into that area, each application node will get notified. It will create the event based on the information in the repository, clear the =event.distribute= and publish the event.</p>

<p>The flow can be described as follows:</p>
<ol>
	<li>Client code generates event using OSGi API, if the =event.distribute= should be set, it is using the ~EventUtil.</li>
	<li>Client code sends the event to the (local) event admin.</li>
	<li>Event admin delivers the event locally.</li>
	<li>Clustering event handler receives the event if =event.distribute= is present
	<ol>
		<li>Event handler adds =event.application= and writes the event to the repository</li>
		<li>Remote repository observers get notified through JCR observation about the new event. They distribute the event locally with the =event.application= (from the node where the event occured first) and cleared =event.distribute=.</li>
	</ol>
	</li>
</ol>


<h3><a name="ApacheSlingEventingandJobHandling-ProcessingJCREvents"></a>Processing JCR Events</h3>

<p>JCR events are environment generated events and therefore are sent by the repository to each node in the cluster. In general, it is advisable to not built the application on the low level repository events but to use application events. Therefore the observer of the JCR event should create an OSGi event based on the changes in the repository. A decision has to be made if the event should be a job or a plain event.</p>

<p>The flow can be described as follows:</p>
<ol>
	<li>Client registers for JCR observation</li>
	<li>JCR notifies the client for changes</li>
	<li>Client generates OSGi event based on the JCR events (the =event.distribute= will not be set), it decides if it sends this event as a job.</li>
	<li>Client code sends the event to the (local) event admin</li>
	<li>Event admin publishes the event locally</li>
	<li>The distribution event handler does not set see the event as the =event.distribute= is not set.</li>
	<li>The job event handler gets the event if it has the job topic
	<ol>
		<li>The job event handler adds the =event.application= property and tries to write the job to the repository
		<ol>
			<li>If no job with the topic and =id= property is in the repository, the event will be written and locked.</li>
			<li>If an event with the topic and =id= property is in the repository then:
			<ol>
				<li>If the =event.application= equals the current application node, the event is set to active (=event:active=) in the repository again and locked</li>
				<li>If the =event.application= does not equal the current application node, the event is not distributed locally.</li>
			</ol>
			</li>
			<li>If the job could be locked in the repository, the job event handler delivers the job locally and synchronously and it unlocks the job and sets =event:active= to false afterwards.</li>
		</ol>
		</li>
	</ol>
	</li>
</ol>


<h3><a name="ApacheSlingEventingandJobHandling-SendingScheduledEvents"></a>Sending Scheduled Events</h3>

<p>Scheduled events are OSGi events that have been created by the environemnt. They are generated on each application node of the cluster through an own scheduler instance. Sending these events works the same as sending events based on JCR events (see above).</p>

<p>In most use cases a scheduler will send job events to ensure that exactly one application node is processing the event.</p>

<h3><a name="ApacheSlingEventingandJobHandling-ReceivingOSGiEvents"></a>Receiving OSGi Events</h3>

<p>If you want to receive OSGi events, you can just follow the specification: receive it via a custom event handler which is registered on bundle start - a filter can be specified as a configuration property of the handler. </p>

<p>As we follow the principle of distributing each event to every registered handler, the handler has to decide if it will process the event. In order to avoid multiple processing of this event in a clustered environment, the event handler should check the =event.application= property. If it is not set, it's a local event and the handler should process the event. If the =event.application= is set, it's a remote event and the handler should not process the event. This is a general rule of thumb - however, it's up to the handler to make its decision either on =event.application= or any other information.</p>

<p>It is advisable to perform the local event check even in a non clustered environment as it makes the migration to a cluster later on much easier and there is nearly no performance overhead caused by the check.</p>

<p>The ~EventUtil class provides an utility method =isLocalEvent(Event)= which checks the existance of the =event.application= property and returns =true= if it is absend.</p>

<h2><a name="ApacheSlingEventingandJobHandling-Scheduler"></a>Scheduler</h2>

<p>Each Sling based application will contain a scheduler service (which is based on the Quartz open source project).</p>

<h2><a name="ApacheSlingEventingandJobHandling-UseCases"></a>Use Cases</h2>

<h3><a name="ApacheSlingEventingandJobHandling-PostProcessing%28BusinessProcesses%29"></a>Post Processing (Business Processes)</h3>

<p>A typical example for post processing (or running a business process) is sending an email or creating thumbnails and extracting meta data from the content (like we do in DAM), which we will discuss here.</p>

<p>An appropriate JCR observer will be registered. This observer detects when new content is put into the repository or when content is changed. In these cases it creates appropriate =CONTENT_ADDED=, =CONTENT_UPDATED= OSGi events from the JCR events. In order to ensure that these actions get processed accordingly, the event is send as a job (with the special job topic, the =topic= and =id= property).</p>

<p>The event admin now delivers these jobs to the registered handlers. The job event handler gets notified and (simplified version) sends the contained event synchronously. One of the handlers for these events is the post processing service in DAM. The job mechanism ensures that exactly one application node is post processing and that the process has to be finished even if the application node dies during execution.</p>

<h2><a name="ApacheSlingEventingandJobHandling-Scheduling"></a>Scheduling</h2>

<p>The scheduler is a service which uses the open source Quartz library. The scheduler has methods to start jobs periodically or with a cron definition. In addition, a service either implementing =java.lang.Runnable= or =org.quartz.job= is started through the whiteboard pattern <b>if</b> it either contains a configuration property =scheduler.expression= or =scheduler.period=. The job is started with the ~PID of the service - if the service has no PID, the configuration property =scheduler.name= must be set.</p>
    </div>
        <div id="commentsSection" class="wiki-content pageSection">
        <div style="float: right;">
            <a href="https://cwiki.apache.org/confluence/users/viewnotifications.action" class="grey">Change Notification Preferences</a>
        </div>
        <a href="https://cwiki.apache.org/confluence/display/SLINGxSITE/Apache+Sling+Eventing+and+Job+Handling">View Online</a>
        |
        <a href="https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=24183351&revisedVersion=2&originalVersion=1">View Changes</a>
                |
        <a href="https://cwiki.apache.org/confluence/display/SLINGxSITE/Apache+Sling+Eventing+and+Job+Handling?showComments=true&amp;showCommentArea=true#addcomment">Add Comment</a>
            </div>
</div>
</div>
</div>
</div>
</body>
</html>

Mime
View raw message