tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Schultz <>
Subject Re: Recommendations for Automating Deployments and then dev, qa, uat, prod testing
Date Mon, 25 Jan 2010 22:30:56 GMT
Hash: SHA1


On 1/23/2010 11:17 AM, Fletcher Cocquyt wrote:
> This process is not perfectly defined and slow - the developer's check all the
> code into SVN and tag the release - ops checks it all out and builds it on the
> tomcat nodes with the dev supplied ant build.xml.  This can take several minutes
> per app per node and if there are any errors the 4-6am window is done.

So, you build the project once for each target node? Why not do a single
build and then copy the complete WAR onto each target node?

Something like this:

$ svn checkout
$ ant test
$ ant war
$ scp mywebapp.war user@node1:
$ scp mywebapp.war user@node2:
$ scp mywebapp.war user@node3:

Of course, you'd need to put the WAR file in the right place, bounce
Tomcat, and possibly run any upgrade scripts required before the webapp
would work properly, but that's the general idea.

> Ideally, the entire release to production process would be such a well defined,
> quick and pre-tested event, than ops would  only be notified if the 4am
> deployment failed and the automated rollback also failed.

It's always a good idea to smoke-test a release, even if it's been
pre-tested, because something stupid like the dev db password being used
in production can really ruin your day if you're not paying attention.

> So questions about the things maven seems to address:
> testing: unit, integration testing - article mentions selenium?
> sensitive data: database passwords (can these be securely handled in maven?)

We have a set of ant scripts that are checked into CVS, but they utilize
a user-specific ~/ file that defines all kinds of things
like installation directories, context paths (i.e. the name of the WAR
file to build), and certain other configuration parameters that are
merged-into the source, etc. at build time.

Thus, in production, we have the production database credentials stored
here as well as other things. These values then don't ever make it into
our source repo, so we don't have to worry about revealing secrets
through that mechanism.

Selenium is a UI testing framework that, frankly, is one of the coolest
things I've ever seen. Its very fragile, though, because you have to
make sure that every screen in your UI includes mark-up to be able to
reference everything, plus every time you make a change, you need to
change your tests. This isn't always the case with code modifications:
you are more likely to change the implementation of a method or class
than to change the interface, and so most tests should pass with no
modification. I would recommend that Selenium-based testing only be done
as part of your develop/test cycle, and not make it a part of the
production build.

For unit testing, I would also say that it should be a part of your
dev/test process, and leave it out of the production build. You could
mandate that tagging the repo for a release requires a complete set of
unit and/or integration tests be run against what will be tagged or
something. Presumably, if it passes tests in dev/test, it will pass
those offline tests in production, too.

Ideally, your testing environment simulates production in every way
except maybe for a handful of service endpoint locations (URLs),
credentials, and, of course, physical/logical environment. The process
for deploying to production should be to take the successfully-tested
WAR file(s) from the testing environment, update the configuration to a
production one, push those WAR files directly into production, then do a
smoke test to make sure that your production configuration was applied
properly. You could even double-check this before the actual deployment
if you don't trust your tools/process.

> Additional, medium-long term goals:
> Standardize the development environment and processes.

We've always been a fan of our ant scripts. In production, this is
essentially the upgrade procedure:

$ cvs checkout
$ ant tomcat-stop
(run database upgrade script)
$ ant install-clean install tomcat-start
(perform smoke tests)

This kind of thing even works in development: all ant targets are
regularly exercised, because they are used in all environments. There's
no special set of procedures for production... it's just like dev/test/demo.

> Leverage the virtual infrastructure we have built with vmware: have the
> developers use standard VM images from templates (ensure consistent JDK, libs
> etc), integrate with lab manager/vApp/VMware Studio concepts.

Heh... if you wanted to, you could rotate VM instances in and out of
production. Your upgrade procedure (for the app servers) could simply be
to switch the IPs of the testing system(s) to the IPs of the production
system(s): instant upgrade, and you know you've already tested everything :)

- -chris
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla -


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message