directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Karasulu <aok...@bellsouth.net>
Subject [ApacheDS][performance] Add, Search, Del throughput
Date Tue, 11 Jul 2006 05:02:14 GMT
Hi folks,


While trying to profile ApacheDS I was playing with tunable cache 
parameters.  Only recently, we made the entry cache and index cache 
sizes tunable so I was curious to see how these effected performance. 
For performance tests I started working with 10,000 entries and tried 
various configurations of the server's memory along with these cache 
sizes.  The key was to determine how fast I can get the server to add, 
delete and return all these entries.


I found that the existing defaults in RC3 were way less than optimal. 
Namely we had far too much cache allocated to indices which were robbing 
the server of memory causing it to have major GCs way to often.  The 
balance between the cache sizes and max heap size is critical.  It makes 
a big difference to performance.


After several configurations I arrived at one which seemed to lead to 
the best performance when using one single threaded client.  The entry 
cache size was set to 10K and the index cache sizes were set to 10 with 
a heap size of 96MB.  Increasing the heap above this value had no 
benefit with this simplified testing scenario.  Of course as we increase 
the number of threads and clients this may change.


Here's the machine's processor:

akarasulu@newton:~$ cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 10
model name      : AMD Athlon(TM) XP
stepping        : 0
cpu MHz         : 2500.561
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 mmx fxsr sse syscall mmxext 3dnowext 3dnow
bogomips        : 5005.28


Here's are the performance stats I got on that machine:

[NOTE: EPS means entries per second]

Add     10K entries           => 1m19.774s (125  EPS)
Search  10K entries (1st try) => 0m12.208s (819  EPS)
Search  10K entries (2nd try) => 0m4.716s  (2120 EPS)
Search  10K entries (3rd try) => 0m4.861s  (2057 EPS)
Del     10K entries           => 0m41.049s (244  EPS)

As you can see the throughput experiment for a single client for these 
simple operations is pretty damn good.  Btw I got better performance 
when I piped the client's output to /dev/null from the ldapXXXX clients: 
I had quoted higher values to Emmanuel over IM.

At ApacheCon EU Emmanuel did a presentation with a different setup. 
First of all the server could not be tuned properly so it had issues 
with memory management and performance would be effected.  Secondly the 
nature of the experiments were different. The common case where a bind, 
simple one entry lookup, followed by an unbind was done.  The number of 
times a server can do this was measured per second.  ApacheDS did not 
perform that well here and some profiling indicates this may be due to 
an overhead in the optimizer.  This optimizer does some calculations 
before beginning a search to increase performance however it has a cost. 
  For simple lookups returning a single entry this may not be optimal. 
I think adding an option to disable the optimizer is a good idea.


Just for a point of reference I decided to try these experiments with 
OpenLDAP.  Ok some word of caution before I reveal these stats.  I 
freaked when I first did a run with OpenLDAP because the performance had 
to be way too low.  So then I started looking into seriously tuning the 
server with the bdb backend.  I wanted to make sure it had enough cache 
to keep all 10K entries in memory as did ApacheDS.  I adjusted these 
values and only got a 30% improvement.  In conclusion I feel that I must 
be doing something wrong.

*Quanah* this is where we may need your expertise!  I'll publish these 
sorry values for now but I'd like to make sure people understand I'm not 
as experienced with tuning OpenLDAP (although I've done it a few times 
in the past). So they may not be figures from the most optimial 
configuration:


Add     10K entries           => 6m13.013s  (27 EPS)
Search  10K entries (2nd try) => 19m14.748s (9  EPS)
Search  10K entries (2nd try) => 13m57.378s (12 EPS)
Search  10K entries (3rd try) => 13m41.407s (12 EPS)
Del     10K entries           => 4m46.699s  (35 EPS)

*Yes these are all in minutes!*

Again to be fair I am piping stdout output to /dev/null.  The following 
commands were used for these respective operations:

[For OpenLDAP running on port 389]
----------------------------------

time ldapadd -h localhost -p 389 -D 'cn=admin,dc=example,dc=com' -x -w 
secret -f add.10k.ldif >> /dev/null

time ldapsearch -h localhost -p 389 -D 'cn=admin,dc=example,dc=com' -x 
-w secret -b 'dc=example,dc=com' 1.1 > /dev/null
*[0]*

time ldapdelete -h localhost -p 389 -D 'cn=admin,dc=example,dc=com' -x 
-w secret -f del.10k.ldif

Check out the following wiki page for more information about these 
tests: *[1]*.  Here you can find the ApacheDS installer which I used for 
these tests: *[2]*.


Alex

-----

*[0]* - We are not even returning whole entries! '1.1' for the attribute 
list only returns the DN of the entry.  I made sure we have presence and 
eq matching indices especially for objectClass since this search 
defaults to an (objectClass=*) filter.  This was done before doing 
anything with the server: namely creating the new bdb database files and 
adding the entries.
*[1]* - http://monretia.notlong.com/
*[2]* - http://dulcaned.notlong.com/



Mime
View raw message