cxf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Kulp <dk...@apache.org>
Subject Re: Faster CXF builds and test ports....
Date Tue, 08 Jun 2010 17:19:26 GMT

OK.   I've gone through pretty much all the system tests and have updated them 
to use more dynamic ports.   There are probably a few stragglers that I missed 
and probably a couple places outside the system tests that also should use 
dynamic ports (or the local transport instead), but for the most part, the 
build now completely works in parallel mode.   

I made one other change: it really only uses the dynamic ports when run from 
Maven.   When run from an IDE like eclipse, it just starts at 9000  and counts 
up.   Basic reason is to make it easier to debug by making it easier to find 
data/traces in Wireshark and such.  

Anyway, previously on my machine( Core i7 820QM, 4 cores, 8 threads), a 
simple:
mvn -Pnochecks
would take about 26 minutes.  (after a previous build or build done in eclipse 
or something so pretty much just runs tests)

With the new stuff, I can do:
mvn -T 12 -Pnochecks
and get:

[INFO] Total time: 9:01.475s (Wall Clock)

Pretty impressive boost.   :-)    It definitely does a good job of keeping the 
processors busy for the first 6 minutes or so.  Then the various modules start 
completing and what's pretty much left is the uncategorized systems tests, the 
ws-* sys tests, and the jaxrs sys tests.  We could even make things faster by 
splitting those up some more to allow more parallelisms, but I'm pretty happy 
with the results right now.    :-)

Dan



On Thursday 03 June 2010 4:43:51 pm Daniel Kulp wrote:
> Actually, I discovered a better way to handle the dynamic ports:
> 
> Since a LARGE LARGE majority of our tests that need ports go through the
> ClientServer infrastructure in testutils, it was fairly easy to add some
> level of dynamic port support to there.    I've now converted all the
> systest/jaxws and systest/databinding tests to use dynamic ports using
> this and it seems to work fairly well.
> 
> Basically, prior to calling the launchServer(...) stuff, the client makes a
> quick call to allocatePort(...) or TestUtils.getPortNumber(...) (the later
> if it doesn't subclass the ClientServerTestBase stuff) which will allocate
> the port and record it based on the information passed in. (I normally use
> the servers class as the name)   In launchServer, the server that is
> forked gets all the allocated ports passed to it as system properties. 
> Thus, when the server needs a port, it also calls the allocatePort thing
> using the same name and it gets back the value that the test produced.
> 
> Also, since everything is stored in system properties, this works well with
> the Spring PropertyPlaceholderConfigurer.   The beans.xml things that
> define the URL's can easily be updated to use the dynamic ports as well. 
> Just add the configurer bean and then use something like:
> 
> address="http://localhost:${testutil.ports.AegisJaxWsTest}/aegisJaxWs"
> 
> to define the location.
> 
> Anyway, as I mentioned, I now have all of jaxws and databinding system
> tests updated to use dynamic ports.   I'll probably tackle the other
> systest things and the http-jetty transport module tomorrow.   I'm not
> sure what else needs to be looked at.  Probably the JMS transport.
> 
> The JMS transport has another issue I'm not sure how to resolve yet.    The
> jms:address extensor in the wsdl can contain the jndi url to the broker
> (jndiURL=tcp://localhost:61500) which would need to be updated somehow as
> well.   I still need to figure that one out.   :-(
> 
> 
> Dan
> 
> On Thursday 03 June 2010 12:36:34 pm Daniel Kulp wrote:
> > As some of you may have noticed, I've been doing some commits to make
> > things work better with the Maven 3 betas.   One reason is to achieve
> > faster builds. My new notebook has 4 cores (8 threads) and 12GB of RAM
> > and I'd like to get to a point where they are useful.  :-)
> > 
> > Maven 3 has a "parallel" mode (abbreviated // )  where it can build non-
> > dependent projects in parallel.  That would allow better use of extra
> > cores and such.   With my latest changes along with an update to the
> > Checkstyle plugin which I committed to Maven yesterday, (needs the
> > snapshot version which isn't deployed yet) I can now build CXF in //
> > mode as long as the tests are off:
> > mvn clean
> > mvn install -Dmaven.test.skip.exec=true -T 12
> > 
> > The // mode cuts about a minute and a half off the install.  (from 6.5
> > minutes down to 5 minutes) which is good.  Subsequent  "mvn
> > -Pfastinstall" builds drop from 43 seconds to 34 seconds.   Also good.
> > 
> > However, the real win will be if I can get the tests running in parallel
> > as well.     The major problem with the tests is the port usage.   We
> > pretty much use ports in the 9000-9100 range for everything with MANY
> > tests using the same ports.   One reason is that we use the ports burned
> > into the wsdl's rather than overriding them as part of the test  cases. 
> >  Thus, I'd like to start going through the tests and get them to
> > override the wsdl specified ports with unique ports.   Now, the question
> > comes: how to allocate the ports.   There are basically two options:
> > 
> > 1) Assign each module a "range" of ports to use and update the tests to
> > use that range.
> > 
> > 2) Use the buildhelper:reserve-network-port maven plugin to try and
> > dynamically allocate some ports and somehow get them passed into the test
> > suites to use.
> > 
> > (1) is definitely easier, but I kind of like (2).   I might try and get
> > the buildhelper thing to allocate a range of like 10 ports per module
> > and save them to a properties file or something and update the testing
> > harness things to load them up to make them avail.   Not really sure
> > yet.
> > 
> > The other trick thing will be the non-http ports we use in the tests.
> > Mostly, that is the JMS broker stuff and the CORBA binding IOR things.
> > Those may also require some extra investigation.
> > 
> > Anyway, just wanted to provide a quick update on the progress.   :-)

-- 
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog

Mime
View raw message