directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Karasulu <aok...@bellsouth.net>
Subject Re: [ApacheDS] Performance testing
Date Fri, 30 Jun 2006 15:27:25 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ole you sure do have a lot of energy :).

...

>> Hmmm I see where you're going.  You want to generate
>> projects that have
>>  a server.xml that can be customized for the test
>> case.  What about
>> having test cases for embedded operation where the
>> server.xml does not
>> come into play?
> 
> There can be 2 separate archetypes for these cases.

...

> But, what if it did'nt have everything we wanted in
> the starting point for this project.
> 
> We just create a maven project using the archetype
> plugin, which is passed as a parameter to the
> archetype:create command and this project creates a
> project that is the starting point for creating other
> projects of a specific type, just like a .java file is
> a means of creating an object of a specific type.

So your archetype creates maven modules that are themselves archetypes?
 Hmmm from below this does not seem to be the case.

> So now we can go ahead and create a project starting
> point (directories and files) that we will create over
> and over again whenever that specific starting point
> is needed.
> 
> So sometimes we want to manipulate the server.xml
> file, 
> 
> I'm just going to interrupt myself here for a
> minute...there should be an outuput directory that
> captures the diff/patch to the server.xml from the
> starting server.xml...ok...continuing

So the archetype changes the initial prepackaged server.xml by asking
the user questions?

> so there's one specific archetype for that type of
> project, and sometimes we just want to configure ADS
> programatically, so we don't need the server.xml, so
> we could make a different archetype for that type of
> project, although since having a directory with
> server.xml in it does not inter with programmtic
> configuration, it might be easier to just explain in
> the documentation that the server.xml is there...and
> that it can be used, or ....  can be used to configure
> ADS.

Ok

> So when testing ADS the developer would run
> mvn archetype:create .... followed by the specific
> parameters for creating an ADS testing project.
> 
> This ensures that everyone testing ADS starts with the
> same inputs.
> 
> Someone mentioned generating LDIF files.
> 
> To do this we could create another input directory
> in the archetype that contained a file with some
> configuration parameters for generating the LDIF.
> 
> That way everyone just has to specify that file for
> apples to apples comparisons.

Following ...

> Similary, there could be a distributed client
> configuration input file that coordinates multiple 
> clients that are hitting ADS, along with a
> corresponding mojo for executing that.

+1

> That file would store things like network
> configuration params, hardware on client(s), hardware
> on ADS server, thus also providing the apple for
> comparison.
> 
>> Thus there should be one
>>> archetype per DS.  So apache would have one, and
>>> OpenLDAP would have another.  
>> Ok I see now.  The OpenLDAP archetype is a bit more
>> difficult to pull
>> off tho since it is C-code and not embeddable or
>> launchable via test
>> cases.  We might have to fork an instance of it.
> 
> Yeah - I just through this into the mix, since one of
> the goals was to compare results with other servers.
> 
> If I had to do it right now I would just run OpenLDAP
> on a "Sterilized" server that I'm also running apache
> ds on and then point OpenLDAP to the configuration
> files in the test directory...which I'm just assuming
> is possible.
> 
> Then code that archetype's mojo to perform testing in
> a way that is as close to the ADS mojo as possible, or
> precisely the same as the ADS mojo, by making perhaps
> using the same mojo executing the same JNDI code on
> both servers.  Thus the only difference would be /
> could be configuration server configuration.
> 

Right there is a distinction between server setup and tests which should
be reusable on different servers.

...

> So we are starting with the same inputs, we can have
> many projects that only do performance testing, but
> change the inputs in various ways, and this is clean
> and easy to see, because all the projects have the
> same structure.

I like this project structure consistency.  I just never thought of
making a new maven project for each kind of test.  But it does make sense.

My ass backwards thinking was stuffing everything into a base JUnit test
case and extending that for a set of performance tests.  Then using a
maven profile to just run the performance tests.  And at times this
won't make sense since we need to bombard the server with serveral
clients.  Meaning the project just launches an LDAP server rather than
running a simple unit test and getting the results.  The test may never
end.  An archetype and plugin does break things down better.

> For instance we might have a single performance
> testing project that gets updated and rechecked into
> subversion, but now someone would have to go back in
> time through subversion to see how inputs have changed
> over time, rather than just comparing each project.

Makes sense.  The maven peeps have each integration test as a separate
maven project module.

> So we would never check out a performance testing
> project, change something, run tests, and check it
> back in, unless we wanted to have that project as a
> starting point for comparison to itself.
> 

Right just generate a new perf testing project, and check that in.  A
new project whenever a new perf test is needed.

...

>> What I'm looking to do is to work with a couple guys
>> here over the
>> course of the next 1-3 weeks to erect a solid
>> framework.  If we can add
>> some of this profiler spice it would be really cool
>> but not a
>> requirement.  FYI I'm a big fan of Yourkit but free
>> tools are much more
>> appealing.
>>
> 
> 
> I played with TPTP for a day and was very impressed by
> it's ability to coordinate agents that could hit ADS
> from multiple clients.
> 
> I think the integration would be cool in terms of
> having a TPTP configuration file that both maven and
> eclipse understand, thus a mojo could coordinate TPTP
> and it could be done through eclipse as well, when
> hands on experimenting is needed. 

Cool.

>>> - Something similar for OpenLDAP...
>>> - Bind this mojo to the maven lifecycle in each
>>> archetype.
>>> - Now anyone can create a quick maven DS
>> performance
>>> testing project, make a few configuration file
>> changes
>>> if that's what's being tested, add unit tests if
>>> necessary, and go!
>> Sounds really fun to do: it's a nice little project
>> and the utility will
>> be high.  I like the idea of having different mojos
>> for each LDAP server
>> that needs to be setup and leaving setup tasks to
>> the mojo/archetype
>> pair.
>>
>>> - The mojo should also produce put the reports in
>> an
>>> output directory somewhere within the site
>> directory,
>>> and this directory should be the same for all DSs.
>> This will be nice to have ... an easily published
>> set of results.
>>
>>> I'll be glad to throw together a sample archetype
>> for
>>> ApacheDS if this sounds interesting.
>> I'm interested in the idea.  I'd love to take a look
>> at it.  My Maven 2
>> archetype skills are non-existent tho.  I need to
>> brush them up.
> 
> 
> I'll throw together a starting baseline asap.
> 
> Here's the quickee on maven archetype development just
> for future reference.
> 
> http://maven.apache.org/guides/mini/guide-creating-archetypes.html

Thanks!

>> SOME NOTES
>> ==========
>>
>>  o [Observation] As Quanah suggested in this thread,
>> we need more than
>> one client to really stress test an LDAP server for
>> total throughput
>> under high concurrency.  This however is one kind of
>> test we will
>> perform: there are also capacity tests as well.  We
>> can still use the
>> recommended tools (SLAMD or another), the partial
>> goal of this effort is
>> to reduce the setup, and running of LDAP servers to
>> be tested in various
>> ways.  We want to automate the setup process with an
>> easy way to
>> create/setup a harness for various LDAP servers with
>> similar testing
>> conditions.
>>
>>  o Why Maven?  It is important to figure out why we
>> (me too btw) want to
>> use Maven.  OK PLEASE DON'T FLAME ME ON THIS GUYS
>> :D.  First off it's
>> our build tool of choice for this project.   
>> Secondly it can run test
>> cases, and fork them if need be with new JVM
>> settings.  It's got a nice
>> plugin architecture and has these things for
>> creating project templates
>> called archetypes.
>>
>> BTW Ole, you caught me by surprise when you
>> suggested using archetypes
>> and plugins (mojos) for this effort.  The idea gives
>> me a good feeling
>> even though I don't know why yet ... there must be
>> many good advantages
>> to this.
>>
>> Alex
> 
> 
> 
> Well - my goal when I started 4 years ago was to be
> able to manufacture software while automating
> everything that can be automated, which I initially
> started doing a ant prototype for, and then said Nuuu
> way - and started looking at Maven 2, which was like
> going from cutting the lawn with scissors to Gasoline
> power Baby!

:) good analogy.

...

> OK - I'll send out a zip of a sample archetype
> creation file along with install instructions for it,
> so that anyone who wants to can try it out.

Perhaps I can setup something in the sandbox for us to play with these
ideas.

Still though I'm open to multiple approaches.

Alex


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEpULdPVto+tI7JJARAguEAJ4rM0tS/mQUQ8ulDotz3qephytO9gCgq8JG
mhRWh1OjWgaYaB1cN2+z9Qk=
=yViQ
-----END PGP SIGNATURE-----

Mime
View raw message