directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ole Ersoy <>
Subject Re: [ApacheDS] Performance testing
Date Fri, 30 Jun 2006 05:25:12 GMT
Yo Yo!

More Inline...

--- Alex Karasulu <> wrote:

> Hash: SHA1
> Ole Ersoy wrote:
> > Hey Alex,
> Hey Ole good to hear from you again.

Yeah - Sorry - been a bit busy proving to the wife
that I'm really not just playing video games on the
computer all day - ... ok
- enough of the shameless self promotion
> > Welcome back from vacation!
> Thanks.  It was too short.
Thank God 4th of July is coming up!

> > - Created maven archetypes specific to performance
> > testing directory servers, with a set of
> directories
> > for DS configuration files.  
> Hmmm I see where you're going.  You want to generate
> projects that have
>  a server.xml that can be customized for the test
> case.  What about
> having test cases for embedded operation where the
> server.xml does not
> come into play?

There can be 2 separate archetypes for these cases.

Just a quick review of the point of the archetype and
some general review.  BTW - I always go for as simple
as possible, just for everyone's benefit...including please forgive me if I'm getting to

The purpose of a project archetype is to provide a
starting point for a project that produces a specific
result.  So there is one archetype for a webapp, one
for writing java code, etc.

Most of us have run mvn archetype:create ....

And then maybe cd'd into the project directory just
created and run mvn eclipse:eclipse.

Now the project can be imported into eclipse, since
the maven eclipse plugin produced all the additional
project files eclipse needs to understand classpath
setting, etc.

So the 2 maven commands produced a project that we can
use as a starting point in eclipse, with default
directories for application source code, test code,
resources, etc.

But, what if it did'nt have everything we wanted in
the starting point for this project.

We just create a maven project using the archetype
plugin, which is passed as a parameter to the
archetype:create command and this project creates a
project that is the starting point for creating other
projects of a specific type, just like a .java file is
a means of creating an object of a specific type.

So now we can go ahead and create a project starting
point (directories and files) that we will create over
and over again whenever that specific starting point
is needed.

So sometimes we want to manipulate the server.xml

I'm just going to interrupt myself here for a
minute...there should be an outuput directory that
captures the diff/patch to the server.xml from the
starting server.xml...ok...continuing

so there's one specific archetype for that type of
project, and sometimes we just want to configure ADS
programatically, so we don't need the server.xml, so
we could make a different archetype for that type of
project, although since having a directory with
server.xml in it does not inter with programmtic
configuration, it might be easier to just explain in
the documentation that the server.xml is there...and
that it can be used, or ....  can be used to configure

So when testing ADS the developer would run
mvn archetype:create .... followed by the specific
parameters for creating an ADS testing project.

This ensures that everyone testing ADS starts with the
same inputs.

Someone mentioned generating LDIF files.

To do this we could create another input directory
in the archetype that contained a file with some
configuration parameters for generating the LDIF.

That way everyone just has to specify that file for
apples to apples comparisons.

A mojo (which is a maven pojo - and a group of mojos
make a maven plugin)
would then run and generate the test data from the
ldif input file.

Similary, there could be a distributed client
configuration input file that coordinates multiple 
clients that are hitting ADS, along with a
corresponding mojo for executing that.

That file would store things like network
configuration params, hardware on client(s), hardware
on ADS server, thus also providing the apple for

> Thus there should be one
> > archetype per DS.  So apache would have one, and
> > OpenLDAP would have another.  
> Ok I see now.  The OpenLDAP archetype is a bit more
> difficult to pull
> off tho since it is C-code and not embeddable or
> launchable via test
> cases.  We might have to fork an instance of it.

Yeah - I just through this into the mix, since one of
the goals was to compare results with other servers.

If I had to do it right now I would just run OpenLDAP
on a "Sterilized" server that I'm also running apache
ds on and then point OpenLDAP to the configuration
files in the test directory...which I'm just assuming
is possible.

Then code that archetype's mojo to perform testing in
a way that is as close to the ADS mojo as possible, or
precisely the same as the ADS mojo, by making perhaps
using the same mojo executing the same JNDI code on
both servers.  Thus the only difference would be /
could be configuration server configuration.

> The configuration files
> > directory should mirror the DS's configuration
> file
> > directory structure.  This isolates the DS's
> > configuration settings per performance test, and
> > provides the initial baseline for everyone to use.
> So by using an archetype you plan to isolate each
> performance test case
> into a separate maven module?

Isolate each performance test case per performance
testing project.  So there could be 3 separate
projects each created using the same starting point,
but maybe different hardware, number of clients
hitting ADS, different ADS configuration, etc.

We're probably saying the same thing here though...

> > - Create another Archetype input directory for the
> > LDIF files. 
> Ok starting to loose you here.  Perhaps my knowledge
> of maven archetypes
> are breaking down now.  Can you elaborate?

This is just so that everyone is loading the same LDIF
file(s) for the test.

So we create the starting point project, and it
already comes with the LDIF files, or a configuration
file for generating an LDIF file(s).

Thus we (and each performance testing project that are
being compared) are working with the same input data.

> > - Write a mojo that embeds ApacheDS, loads the
> > configuration files, and then runs all the tests
> on
> > the specific server instance, and then generates
> > reports.  
> Yeah this I understand.  We can do this pretty
> easily.
> This way each instance of this archetype is
> > running the DS in an isolated manner.  
> Hmmm again loosing you here on the meaning or use of
> the m2 archetype.

So we are starting with the same inputs, we can have
many projects that only do performance testing, but
change the inputs in various ways, and this is clean
and easy to see, because all the projects have the
same structure.

For instance we might have a single performance
testing project that gets updated and rechecked into
subversion, but now someone would have to go back in
time through subversion to see how inputs have changed
over time, rather than just comparing each project.

So we would never check out a performance testing
project, change something, run tests, and check it
back in, unless we wanted to have that project as a
starting point for comparison to itself.

> Would also be
> > cool to integrate this stuff with Eclipse TPTP,
> using
> > something similar to the maven eclipse
> plugin...that
> > way all the results are cross compatible with TPTP
> and
> > can be opened up in an eclipse TPTP
> project...maybe in
> > version 2 of the mojo...or mojo plugin set.
> Is this the Eclipse profiling stuff?


> integration would be pretty
> tight.  Perhaps as you suggest a bit overkill for a
> first round.

Definitely...although very nice to have for the round
where distributed client testing starts happening, or
for the round where someone wants to be able to
experiment with the tesing in eclipse....doing on the
fly type testing.

> What I'm looking to do is to work with a couple guys
> here over the
> course of the next 1-3 weeks to erect a solid
> framework.  If we can add
> some of this profiler spice it would be really cool
> but not a
> requirement.  FYI I'm a big fan of Yourkit but free
> tools are much more
> appealing.

I played with TPTP for a day and was very impressed by
it's ability to coordinate agents that could hit ADS
from multiple clients.

I think the integration would be cool in terms of
having a TPTP configuration file that both maven and
eclipse understand, thus a mojo could coordinate TPTP
and it could be done through eclipse as well, when
hands on experimenting is needed. 

> > - Something similar for OpenLDAP...
> > - Bind this mojo to the maven lifecycle in each
> > archetype.
> > - Now anyone can create a quick maven DS
> performance
> > testing project, make a few configuration file
> changes
> > if that's what's being tested, add unit tests if
> > necessary, and go!
> Sounds really fun to do: it's a nice little project
> and the utility will
> be high.  I like the idea of having different mojos
> for each LDAP server
> that needs to be setup and leaving setup tasks to
> the mojo/archetype
> pair.
> > - The mojo should also produce put the reports in
> an
> > output directory somewhere within the site
> directory,
> > and this directory should be the same for all DSs.
> This will be nice to have ... an easily published
> set of results.
> > I'll be glad to throw together a sample archetype
> for
> > ApacheDS if this sounds interesting.
> I'm interested in the idea.  I'd love to take a look
> at it.  My Maven 2
> archetype skills are non-existent tho.  I need to
> brush them up.

I'll throw together a starting baseline asap.

Here's the quickee on maven archetype development just
for future reference.

> > Coding the mojo should also be fairly
> straightforward.
> Yeah I agree plugins are a cake walk in m2.
> ==========
>  o [Observation] As Quanah suggested in this thread,
> we need more than
> one client to really stress test an LDAP server for
> total throughput
> under high concurrency.  This however is one kind of
> test we will
> perform: there are also capacity tests as well.  We
> can still use the
> recommended tools (SLAMD or another), the partial
> goal of this effort is
> to reduce the setup, and running of LDAP servers to
> be tested in various
> ways.  We want to automate the setup process with an
> easy way to
> create/setup a harness for various LDAP servers with
> similar testing
> conditions.
>  o Why Maven?  It is important to figure out why we
> (me too btw) want to
> :D.  First off it's
> our build tool of choice for this project.   
> Secondly it can run test
> cases, and fork them if need be with new JVM
> settings.  It's got a nice
> plugin architecture and has these things for
> creating project templates
> called archetypes.
> BTW Ole, you caught me by surprise when you
> suggested using archetypes
> and plugins (mojos) for this effort.  The idea gives
> me a good feeling
> even though I don't know why yet ... there must be
> many good advantages
> to this.
> Alex

Well - my goal when I started 4 years ago was to be
able to manufacture software while automating
everything that can be automated, which I initially
started doing a ant prototype for, and then said Nuuu
way - and started looking at Maven 2, which was like
going from cutting the lawn with scissors to Gasoline
power Baby!

So yeah - working with a standardized structure and
standardized ways, that don't limit flexibility, in
working on that structure is fantastic, especially
since it's becoming a standard build tool/software
manufacturing tool of choice in open source and

For instance right now I'm creating archetypes that
automate most of the work involved in creating Java
Server Faces components, thus in the end all I have to
do in provide a single configuration file, and it
automatically generates all the JSF files, including
configuration files, component java files, renderer
files, projects for deploying the testing of the
component to Tomcat, testing, packaging, etc. etc.

And all this is really easy to do in an iterative way
with Maven's multi project capability.

Which reminds me - this article is really great in
case anyone wants to read up on the type of stuff I
just mentioned:

OK - I'll send out a zip of a sample archetype
creation file along with install instructions for it,
so that anyone who wants to can try it out.

- Ole

> > 
> > - Write a maven mojo that 
> > 
> > --- Alex Karasulu <> wrote:
> > 
> > Need Benchmarking/Profiling Perf Suite (BPPS) for
> > ApacheDS
> > 
> >>
> > I just sat down in front of my machine and wanted
> to
> > run some
> > performance tests on the server.  These questions
> > came up:
> > 
> > o What tests do I run?
> > o What LDIF should I import first before running
> > those tests?
> > o Which version of the server should I run these
> > tests against?
> > o Can I run these tests against other LDAP
> servers?
=== message truncated ===

Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 

View raw message