Return-Path: X-Original-To: apmail-directory-dev-archive@www.apache.org Delivered-To: apmail-directory-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 09D6E96F0 for ; Thu, 12 Apr 2012 14:15:03 +0000 (UTC) Received: (qmail 21954 invoked by uid 500); 12 Apr 2012 14:15:03 -0000 Delivered-To: apmail-directory-dev-archive@directory.apache.org Received: (qmail 21784 invoked by uid 500); 12 Apr 2012 14:15:02 -0000 Mailing-List: contact dev-help@directory.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Apache Directory Developers List" Delivered-To: mailing list dev@directory.apache.org Received: (qmail 21777 invoked by uid 99); 12 Apr 2012 14:15:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Apr 2012 14:15:02 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of akarasulu@gmail.com designates 209.85.212.178 as permitted sender) Received: from [209.85.212.178] (HELO mail-wi0-f178.google.com) (209.85.212.178) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Apr 2012 14:14:56 +0000 Received: by wibhq7 with SMTP id hq7so2014649wib.1 for ; Thu, 12 Apr 2012 07:14:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=rviaMEHu7e9F0L8DtWsEDJR8ioKiPwySef+/iJq7uiU=; b=h/OafbCino4xArP6ktGw8qo3Pf0nnvYiYwS3L007gsamraJPReCvFJFAJRrQ/hsPzN dFf7ss8O7UyjFTx2g46ZcUfGTF+ixyXzMF6hAd+IxwFm8iuI5TVS4KDTDmPO+PU0kRCG 8s4/S5W5xzMX7ueOGQvXhOlABSKbRPmymp3cOzMylFTif7cGFTE1I5pCQJcIg/f8FH9G m+pTSVO0dnnHrbKMk1Y+tq/9HFXjoTl3pBaCmQwbROd4wRf7+VRE+JtJuTYGGQT4h+Ml tHBPJef5oY3u/RxV8c60YPOcYspnrE+H0erZhTUEvE91pTXWxfitMQf1K6BbsiEnc+ZY jEyw== MIME-Version: 1.0 Received: by 10.216.136.202 with SMTP id w52mr1619462wei.109.1334240074497; Thu, 12 Apr 2012 07:14:34 -0700 (PDT) Sender: akarasulu@gmail.com Received: by 10.180.126.97 with HTTP; Thu, 12 Apr 2012 07:14:34 -0700 (PDT) In-Reply-To: <4F86DA25.7020906@gmail.com> References: <4F858163.2000804@gmail.com> <4F86DA25.7020906@gmail.com> Date: Thu, 12 Apr 2012 17:14:34 +0300 X-Google-Sender-Auth: hajt6f8PK5yRrBjDHLh0wkuWGrE Message-ID: Subject: Re: [index] OneLevelIndex removal From: Alex Karasulu To: Apache Directory Developers List , elecharny@apache.org Content-Type: multipart/alternative; boundary=0016e6d55a38af7d4604bd7bf94b X-Virus-Checked: Checked by ClamAV on apache.org --0016e6d55a38af7d4604bd7bf94b Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Thu, Apr 12, 2012 at 4:35 PM, Emmanuel L=E9charny w= rote: > Forgot to reply to this mail, which raises interesting points. > > More inside. > > Le 4/11/12 10:38 PM, Alex Karasulu a =E9crit : > >> On Wed, Apr 11, 2012 at 4:04 PM, Emmanuel L=E9charny** >> wrote: >> >> I think we should add some mechanism in the server to check that >>> automatically, to avoid doing it by hand (there are hundreds of tests t= o >>> check...). One solution would be to keep a track of every cursor >>> construction in a HashMap, and to remove them when the cursor is closed= . >>> The remaining cursors are likely not closed. >>> >> >> It would be nice to have a Cursor monitor that every opened Cursor >> registers with but this needs to happen automatically. Then when out of >> the >> creation scope the Cursor is expected to be closed and if not this is >> handled automatically. However does creation scope work well since >> sometimes we create Cursors and pass them up? >> > We do have a monitor, which is currently used to check that the cursor is > not closed when we try to use it. We certainly can use this monitor for > more than just checking such thing. > > Now, the pb is that the scope is not as easy to determinate than for a > variable in Java. For instance, if we consider persistent searches, or > paged searches, or even an abandonned search request, the scope is pretty > wide... > > Though we can have a set of rules that help us to close the cursor > automatically : > - if we get an exception during a SearchRequest, then the cursors must be > closed immediately. As soon as we store the cursors into the SearchContex= t, > this is pretty easy to do > - an AbandonRequest will close the cursor automatically too (getting the > cursor from the abandonned request) > - when we process the SearchResultDone, we can also close the cursor for > the current search request (this work for PagedSearch too) > - for pagedSearch, if the user reset the search by sending 0 as the > expected number of entries to return, then the cursor will be freed > - for persistent searches, as it will be closed by an unbind or an abando= n > request, we are fine > - when a client unbinds, then all the pending cursors will be closed. > > All in all, we have everything needed to close the cursors automatically, > assuming we keep all the cursors into the session. > > These are really great suggestions and make the ideas I tried to express really tangible. Thanks for it Emmanuel. One technical point, we need to make Cursor close() operations idempotent if they are not already - meaning if we close a second time this should not cause an exception or change the outcome. > On the client side, this is another issue... As cursors are created by th= e > client code, we have no easy way to determinate when we should close the > cursors, except when the connection is closed or an abandon request/unbin= d > request is sent. Of course, when the server returns a searchResultDone we > could also close the cursor. Remains the situations where the client has > fetched some entries (but not all), and haven't unbind nor abandonned the > search. > > I think the aspect for automatic closing of cursors is left to be managed inside the server even though the API overlaps here. > In any case, this is less critical as we don't have to deal with the txn > layer. The client will just blow away with some nasty OOM sooner or > later... but this is not worse than what we get with NamingEnumeration in > JNDI, nah ? > > Yup +1 > Have I covered all the server options ? Or did I miss something ? > > >> This sounds like something that can be handled nicely using an aspect >> oriented solution. Now these things are heavy if you use AspectJ or >> something like that but other simpler solutions exist to bytecode splice >> compiled code to automatically handle these things. Maybe our past >> experiences with Aspects might make us reconsider. >> > A bit overkilling, IMO? > > I'm feeling the same but thought it should be just put out there. However we can achieve the same results perhaps with code or using a lighter mechanism with Proxy's via CGlib or something similar. These are just raw thought dumps so it's not a we SHOULD recommendation. Something to think about. --=20 Best Regards, -- Alex --0016e6d55a38af7d4604bd7bf94b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

On Thu, Apr 12, 2012 at 4:35 PM, Emmanue= l L=E9charny <e= lecharny@gmail.com> wrote:
Forgot to reply to this mail, which raises interesting points.

More inside.

Le 4/11/12 10:38 PM, Alex Karasulu a =E9crit :
On Wed, Apr 11, 2012 at 4:04 PM, Emmanuel L=E9charny<elecharny@gmail.com>wro= te:

I think we should add some mechanism in the server to check that
automatically, to avoid doing it by hand (there are hundreds of tests to check...). One solution would be to keep a track of every cursor
construction in a HashMap, and to remove them when the cursor is closed. The remaining cursors are likely not closed.

It would be nice to have a Cursor monitor that every opened Cursor
registers with but this needs to happen automatically. Then when out of the=
creation scope the Cursor is expected to be closed and if not this is
handled automatically. However does creation scope work well since
sometimes we create Cursors and pass them up?
We do have a monitor, which is currently used to check that the cursor is n= ot closed when we try to use it. We certainly can use this monitor for more= than just checking such thing.

Now, the pb is that the scope is not as easy to determinate than for a vari= able in Java. For instance, if we consider persistent searches, or paged se= arches, or even an abandonned search request, the scope is pretty wide...
Though we can have a set of rules that help us to close the cursor automati= cally :
- if we get an exception during a SearchRequest, then the cursors must be c= losed immediately. As soon as we store the cursors into the SearchContext, = this is pretty easy to do
- an AbandonRequest will close the cursor automatically too (getting the cu= rsor from the abandonned request)
- when we process the SearchResultDone, we can also close the cursor for th= e current search request (this work for PagedSearch too)
- for pagedSearch, if the user reset the search by sending 0 as the expecte= d number of entries to return, then the cursor will be freed
- for persistent searches, as it will be closed by an unbind or an abandon = request, we are fine
- when a client unbinds, then all the pending cursors will be closed.

All in all, we have everything needed to close the cursors automatically, a= ssuming we keep all the cursors into the session.


These are really great suggestions and= make the ideas I tried to express really tangible. Thanks for it Emmanuel.=

One technical point, we need to make Cursor close= () operations idempotent if they are not already - meaning if we close a se= cond time this should not cause an exception or change the outcome.
=A0
On the client side, this is another issue... As cursors are created by the = client code, we have no easy way to determinate when we should close the cu= rsors, except when the connection is closed or an abandon request/unbind re= quest is sent. Of course, when the server returns a searchResultDone we cou= ld also close the cursor. Remains the situations where the client has fetch= ed some entries (but not all), and haven't unbind nor abandonned the se= arch.


I think the aspect for automatic closi= ng of cursors is left to be managed inside the server even though the API o= verlaps here.=A0
=A0
In any case, this is less critical as we don't have to deal with the tx= n layer. The client will just blow away with some nasty OOM sooner or later= ... but this is not worse than what we get with NamingEnumeration in JNDI, = nah ?


Yup +1
=A0
Have I covered all the server options ? Or did I miss something ?


This sounds like something that can be handled nicely using an aspect
oriented solution. Now these things are heavy if you use AspectJ or
something like that but other simpler solutions exist to bytecode splice compiled code to automatically handle these things. Maybe our past
experiences with Aspects might make us reconsider.
A bit overkilling, IMO?


I'm feeling the same but thought i= t should be just put out there. However we can achieve the same results per= haps with code or using a lighter mechanism with Proxy's via CGlib or s= omething similar. These are just raw thought dumps so it's not a we SHO= ULD recommendation. Something to think about.

--
Best Regards,
-- Alex

--0016e6d55a38af7d4604bd7bf94b--