Mailing-List: contact dev-help@directory.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Apache Directory Developers List" <dev@directory.apache.org>
Received-SPF: neutral (nike.apache.org: local policy)
Message-ID: <20081217141918.wsqkv4qq0ww8g4ok@www.seelmann-it.de>
Date: Wed, 17 Dec 2008 14:19:18 +0100
From: Stefan Seelmann <seelmann@apache.org>
To: dev@directory.apache.org
Subject: Re: [PagedSearch] Cleanup strategy
References: <4948D894.8030408@nextury.com>
In-Reply-To: <4948D894.8030408@nextury.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset=ISO-8859-1;
	DelSp="Yes";
	format="flowed"
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
User-Agent: Internet Messaging Program (IMP) H3 (4.1.3)


Quoting Emmanuel Lecharny <elecharny@gmail.com>:

> Hi,
>
> this is the last step to handle the paged search request : cleaning the
> remaining paged search when they are not used.
>
> The problem is that a client can launch a paged search, get some few
> pages, and then stop requesting the remaining pages, without closing
> the request. Obviously, this is bad, bad, bad, but we won't be able to
> spank all the clients ;)
>
> The created paged search context will then remain until the session is
> closed, keeping all the associated data (cursors, and such) in memory.
> Plus the client may launch more than one paged search request, leading
> to a quick OOM...
>
> There are a few strategy we can implement to avoid such a problem :
> 1) first, we can limit the number of paged search request per session.
> I guess that a session with more than  5 or 10 opened paged search is
> very unlikely to occur. I suggest that we limit the number of opened
> paged searches to 5 atm (of course, this should be configurable), and
> unlimited for the administartor. When we reach this number, then the
> oldest paged search will be removed from the session.

Maybe a per-user limit would also make sense. A user could open =20
multiple sessions in order to start more paged searches. Also a global =20
limit could make sense. But we should keep it simple ;-)

Another question is how to define the "oldest" paged search. Do we use
- either the timestamp when the initial search were started and the =20
first page was requested
- or the timestamp when the last page was requested?

> 2) we can also manage a timeout for the paged search. That mean a
> thread will be in charge of removing all the time-outed paged search
> across the sessions.

Do we already have another thread for clean-up stuff? Maybe we could =20
plug into such an existing thread.

If the clean-up isn't too time consuming maybe we could do such checks =20
in other requests. e.g. each time when a paged search request comes in =20
we could check that. This would avoid an extra thread.

> 3) We can also wait until the session dies, and cleanup the associated
> paged search.
>
> In any case, we must do something, as the cursors _have_ to be closed.
>
> Does those 3 proposals make sense to you? Did I forgot something? Do
> you see a better strategy ?

The server also implements persistent search. Are there similar =20
problems with open resources? Maybe we can look how it is solved there?

Regards,
Stefan