While implementing the Cursor pattern in the JDBM based partition, some interesting ideas came to mind regarding the new LDAP Client API we've been working on.  Up to now we have simply defined what Entries look like and have called it the Entry API instead.  Eventually this will grow into a full client as we combine more of the pieces together.  JNDI is a less than optimal API for LDAP.

Cursors are special, in contrast to the NamingEnumerations we use, because they can be positioned after creation.  Cursors can be positioned using the beforeFirst(), first(), last() and afterLast() methods at any point during their use.  Furthermore, Cursors are bidirectional; they can be traversed in both directions with calls to next() and previous() to navigate results.  Callers can advance a Cursor forward or reverse at any time after creation.

The Partition interface in ApacheDS will soon expose search results by returning a Cursor<ServerEntry> instead of a NamingEnumeration.  Depending on the filter used, this is a composite Cursor which leverages partition indices to position itself without having to buffer results.  This allows the server to pull entries satisfying the search scope and filter one at a time, and return it to the client without massive latency, or memory consumption.  It also means we can process many more concurrent requests as well as process single requests faster.  In addition a resultant search Cursor can be advanced or position using the methods described above just by having nested Cursors based on indicies advance or jump to the appropriate Index positions.  We already have some of these footprint benefits with NamingEnumerations, however the positioning and advancing capabilities are not present with NamingEnumerations.

During the course of this work, I questioned whether or not client side Cursors would be possible.  Perhaps not under the current protocol without some controls or extended operations.  Technical barriers in the protocol aside, I started to dream about how this neat capability could impact clients.  With it, clients can advance or position themselves over the results of a search as they like.  Clients may even be able to freeze a search in progress, by having the server tuck away the server side Cursor's state, to be revisited later.  For lack of terms I've likened this to a form of asynchronous bidirectional LDAP search. This would eliminate the need to bother with paging controls.  It could even be used to eliminate the thread per search problem associated with persistent search.  OK, let me stop dreaming and start looking at reality so we can determine if this is even a possibility.

So these characteristics of a Cursor have a profound impact on the semantics of a search operation - not talking about the protocol yet.  I'm referring to search as seen from the perspective of client callers using the Cursor: the front end.  As stated search operations can be initiated and shelved to persist the state of the search by tucking away the Cursor in the connection session.  A Cursor for a search will automatically track it's position. 

However the protocol imposes some limitations on being able to leverage these capabilities across the network on an LDAP client.  A search request begins the search, and entry responses are received from the server, until the server returns a search response done operation which  signals the end of the search operation.  During this sequence, without creative extended operations, or controls, there's little the client can do to influence the entries returned by the server or throttle the rate of return.  Of course size and time limits can be set on the search request but after issuing the search, these cannot be altered.  Because the LdapMessage envelop contains a messageId, and all responses contain the messageId of the request they correspond to, the protocol allows for multiple requests to be issued in parallel.  Even if client API's do not allow for it, this is certainly possible.

Although I've long forgotten how the paging control works exactly, I still have a rough idea: forgive me for my laziness and if I'm missing something.  A control specifies some number of results to return per page, and the server complies by limiting the search to that number then capping off the search operation with a search result done.  Cookies in the request and response controls are used to track the progress, so another search request for the next page returns the next page rather than initiating the search from the start.  This breaks a big search up into many smaller search requests.  This way the client has the ability to intervene in what otherwise would be a long flood of results in a single large search operation.  If this page control could also convey information about positioning, and directionality, along with a page size set to 1, we could implement client side Cursors with the same capabilities they posses on the server.  Paging search results effectively has the server tucking away the search Cursor state into the client session and pulling it out again to continue.  This is how we would implement this control today (that is if anyone gets the time to do so :) ).

Persistent search is unrelated however I'd like to explore whether or not there's some possible synergy/relationship between it and paging.  (Persistent search IMHO is poorly implemented but we can deal with it). It is intended for receiving change notifications.  A persistent search control is issued on a search request which may return a bunch of entry responses for the filter if requested, but the most notable thing is that the search does not end with a search result done response.  The operation persists to return entries satisfying the filter along with a response control containing the change type when entries satisfying the filter change accordingly.  Clients usually need to assign a thread to listen for such responses.  Smart clients will use a single thread instead of one per persistent search.  Even smarter clients will use a single thread to listen for search responses on persistent search requests and for unsolicited notifications.  Regardless once the persistent search request is issued, there's no way the client can stop it until size and time limits are reached.  These are parameters in the control sent on the search request.  Of course persistent search requests can use a size limit of 1 and clients can request another persistent search after an event is received to have more control.  Regardless a change many not occur for a while in which case fine tuning with the time limit will help.

It's a bit crazy to think what would happen if both these controls are used together on the same search request.  I guess it all depends on the server implementation in the end.  Not sure if anyone would even want to do this.  Regardless of the pain it would entail, I think this situation can be managed to work in the server.  Now where does this lead us tho?  Perhaps the Cursor interface could be enhanced to support a listener to asynchronously notify users of changes to the underlying results.  The Cursor can then be reset (or just repositioned).  Of course this is all presuming the Cursor was created to traverse results.  Instead a Cursor might just be created to iterate over only changes, but does this make sense?  Whatever the answer, at least we can know when underlying results have changed to invalidate the Cursor on the client.   This is probably a bullsh*t idea but it's entertaining to think about.

BTW change notifications are probably best implemented as a combination of search and extended operations through unsolicited notifications.   The client issues a search request with a control similar to the persistent search request control.  Instead of 'persisting' the search, the search returns immediately with a search result done response using a result code to indicate whether or not the server will honor the request to be notified of changes.  Then the client is done registering the request to be notified.  Whenever the server detects changes that satisfy the changeType, scope and filter of the notification registration, it sends an unsolicited extended response to the client.  The payload carries information about the change which took place similar to the way search entry results do with the response control of persistent search.   The client can issue a deregistration message to the server to stop receiving these notifications using an ExtendedRequest.  The server would respond to this with an ExtendedResponse.  IMO this is a much better mechanism with full control over the subscription and notification process.