directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kiran Ayyagari <kayyag...@apache.org>
Subject Re: Attribute filtering handling...
Date Thu, 13 Dec 2012 11:02:49 GMT
On Thu, Dec 13, 2012 at 2:31 PM, Emmanuel Lécharny <elecharny@gmail.com>wrote:

> Le 12/11/12 2:09 PM, Kiran Ayyagari a écrit :
> > On Tue, Dec 11, 2012 at 4:11 PM, Emmanuel Lécharny <elecharny@gmail.com
> >wrote:
> >
> >> Le 12/11/12 11:02 AM, Kiran Ayyagari a écrit :
> >>> On Tue, Dec 11, 2012 at 3:23 PM, Emmanuel Lécharny <
> elecharny@gmail.com
> >>> wrote:
> >>>
> >>>> But, and there is a big but, when processing a search request, we do
a
> >>>> direct lookup from the backend, and we are not going through this
> chain.
> >>>> What happens is that we are building a list of filters, and we apply
> >>>> this list of filters on every entry fetched from the backend doing a
> >>>> direct lookup. So the filter added by the SchemaInterceptor should
> also
> >>>> do this filtering.
> >>>>
> >>>> as the filtering is only applicable for search(and lookup) why not
> apply
> >>> this logic in the cursor
> >>> after evaluating an entry, I know that this filtering logic was
> scattered
> >>> before but with all that
> >>> moved to a utility class I think using that inside the cursor is the
> >> right
> >>> thing rather than doing it in an
> >>> interceptor
> >> Because Lookup does not use a cursor at all :/ OTOH, cursors are based
> >> on lookup (but not the same as for the lookup() operation.
>
> but unlike in lookup(LookupOperationContext) we don't need to filter
> anything in lookup(id) method
>
> Yep, true.
>
> Btw, I'm done with the Lookup Cleanup (well, I hope).
>
> We don't filter the attributes in the backend anymore, it's all done in
> the SchemaInterceptor.
>
> In the process, I discovered that a lookup going throug the ACI
> interceptor was not going through the chain, as we were doing a direct
> fetch from the backend, bypassing *all* the interceptors :/
> That says two things :
> o we don't have a decent test coverage when it comes to ACI
> o it's not because some code is 7 years old that it's bug free ! :)
>
> As a direct consequence of the cleanup, a lookup operation is now 10%
> faster. Some numbers :
> - a getRootDSE, no filtering : 242 000 lookup/s
> - a lookup on rootDSE, all user attrs (*) : 148 544 lookup/s
> - a lookup on rootDSE, all attributes (*, +) : 145 243 lookup/s
> - a lookup on an entry, all user attributes (*) : 144 906 lookup/s
> - a lookup on an entry, all attributes (*, +) : 140 154 lookup/s
> - a lookup on an entry, all atributes, with ACI (*, +)  : 147 362 lookup/s
>
> 73% of the time is used to create a clone of the entry we will return,
> 10% to fetch the entry ID from the backend using the lookup DN, and 15%
> to filter the attributes. It's pretty optimal at this point with 98% of
> the processing done in areas we barelly can optimize :)
>
> great, thanks for the heads up

> One potential optimization at this point would be to add a parallel data
> structure to the entry we fetch, containing flags set to true for
> Attributes we want to return and false for those that are filtered
> outout. That would spare us the cost of cloning the whole entry, a very
> costly operation. It will impact the way we write the response in the
> encoder, but I'm quite sure that the gain could be substancial...
>
> I still have to check that lookups on subschemaSubentry is ok, and check
> the lookup through the ACI subsystem.
>
> The next step will be to cleanup the search.
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>


-- 
Kiran Ayyagari
http://keydap.com

Mime
View raw message