geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Jencks <david_jen...@yahoo.com>
Subject Re: Speeding up G3.0 build time
Date Wed, 22 Jun 2011 21:49:23 GMT
I've had disk thrashing up my build times from ~30 min to ~2 hours, but never this much.  I
have no evidence but would suspect a DNS problem resulting in multiple DNS timeouts for each
module.

Do you have a more-memory-endowed machine on this network you could try a build on to see
if it's faster?

thanks
david jencks

On Jun 22, 2011, at 1:32 PM, Russell E Glaue wrote:

> On 06/22/2011 01:54 PM, Kevan Miller wrote:
>> 
>> On Jun 22, 2011, at 1:27 PM, Russell E Glaue wrote:
>> 
>>> On 06/22/2011 12:02 PM, Kevan Miller wrote:
>>>> 
>>>> On Jun 22, 2011, at 12:09 PM, Russell E Glaue wrote:
>> 
>> <snip>
>> 
>>>> 
>>>>> 
>>>>> I am using a very basic build procedure.
>>>>> 1. checkout G3 trunk
>>>>> 2. JAVA_HOME="/path/to/jdk/1.6.0_22"
>>>>> 3. download and install a fresh maven 2.2.1
>>>>> 4. MAVEN_OPTS="-Xmx968m -XX:MaxPermSize=500m -XX:ReservedCodeCacheSize=64m"
>>>> 
>>>> My settings are similar. I don't specify '-XX:ReservedCodeCacheSize=64m'.
>>>> 
>>>> I run with the following. I know there are others running with larger max
heaps. I haven't :
>>>> 
>>>> MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError"
> 
> Without the "-XX:ReservedCodeCacheSize=64m" option in MAVEN_OPTS I get this
> OutOfMemory error:
> -
> [INFO] ------------------------------------------------------------------------
> [INFO] Building Geronimo Build Support :: Plugin 3.0-SNAPSHOT
> [INFO] ------------------------------------------------------------------------
> ...<snip>...
> [INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) @
> buildsupport-maven-plugin ---
> [INFO] Compiling 3 source files to
> /data/geronimo/asf-geronimo-server-trunk/framework/buildsupport/buildsupport-maven-plugin/target/classes
> [INFO]
> [INFO] --- gmaven-plugin:1.2:compile (default) @ buildsupport-maven-plugin ---
> ---------------------------------------------------
> constituent[0]: file:/usr/local/apache-maven-3.0.3/lib/maven-embedder-3.0.3.jar
> ...<snip>...
> constituent[30]: file:/usr/local/apache-maven-3.0.3/lib/maven-compat-3.0.3.jar
> ---------------------------------------------------
> Exception in thread "main" java.lang.VirtualMachineError: out of space in
> CodeCache for adapters
>        at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:212)
>        at
> org.codehaus.groovy.ast.builder.AstBuilderTransformation.visit(AstBuilderTransformation.groovy:62)
>        at
> org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:268)
>        at
> org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:799)
>        at
> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:464)
>        at
> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:443)
>        at
> org.codehaus.gmaven.runtime.v1_6.ClassCompilerFeature$ClassCompilerImpl.compile(ClassCompilerFeature.java:159)
>        at
> org.codehaus.gmaven.plugin.compile.AbstractCompileMojo.compile(AbstractCompileMojo.java:200)
>        at
> org.codehaus.gmaven.plugin.compile.AbstractCompileMojo.process(AbstractCompileMojo.java:164)
>        at
> org.codehaus.gmaven.plugin.ComponentMojoSupport.doExecute(ComponentMojoSupport.java:60)
>        at org.codehaus.gmaven.plugin.MojoSupport.execute(MojoSupport.java:69)
>        at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
>        at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
>        at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> -
> 
> With the "-XX:ReservedCodeCacheSize=64m" flag, I do not get this error.
> 
> 
>>> 
>>> The server I am build on has 1024MB of memory total, and has a dual dual-core
>>> Intel(R) XEON(TM) CPU 2.00GHz with 512KB cache.
>>> 
>>> I have tried many JVM configurations, and the one like mine and yours is the
>>> best one I have found. For me, a JVM configuration with less than 400m for
>>> MaxPermSize always ended with an OutOfMemory error on PermGen space.
>>> 
>>> The Geronimo dev wiki suggests 200m for MaxPermSize, which never worked for me
>>> for trunk. I always assumed it was because of all the snapshot downloading the
>>> build process has to do. Even after the 2nd or 3rd build, 200m was never enough.
>> 
>> Right. So, MaxPermSize=200m is definitely out-of-date. There is/may be some bugs
that are increasing this value. However, not really worth our while to pursue at the moment.
I don't think MaxPermSize=500m is too prohibitive. More important things to do, ATM. 
>> 
>> That said -- I'm building on a machine with 8 gigs of real memory. So, 1024MB of
real memory seems very small to me. And I suspect that this is the source of your problem...
A max heap of 1024MB and MaxPermSize of 512m means your system may be spending a lot of time
swapping virtual memory. 
> 
> So wrote what are your JVM Options set at:
> MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError"
> Mine are:
> MAVEN_OPTS="-Xmx968m -XX:MaxPermSize=500m -XX:ReservedCodeCacheSize=64m"
> 
> The machine I am using has 1024MB of real memory and is dedicated to building
> geronimo (and it is not a virtual server). I set the heap max to 968m to keep
> the memory usage from peaking at 100% of system capacity.
> 
> But there is a chance you may be right about the swaping. the machine runs at
> 100% real memory usage (expected) and about ~50% swap.
> 
> 
> I thinking recommending a minimum amount of real memory a machine should have to
> adequately compile G3 in a reasonable amount of time should be suggested.
> What would you say the minimum recommendation should be? For Windows and Linux?
> 2GB for Windows and 1.5GB for Linux, assuming the machine is dedicated to compiling?
> 
>> 
>>> 
>>>> 
>>>>> 5. mvn clean install
>>>>> 6. 13 to 39 hours later == BUILD SUCCESS
>>>>> 
>>>>> 
>>>>> From observing the build process, it would seem the most used (seemingly
paused)
>>>>> time is right before the build process moves on to the next plugin or
assembly.
>>>>> 
>>>>> This "seemingly paused time" after the assembly I assume is the packaging
>>>>> process, because the resulting output is a huge collection.
>>>>> 
>>>>> [INFO] Geronimo Assemblies ............................... SUCCESS [3.995s]
>>>>> [INFO] Geronimo Assemblies :: Minimal + Jetty8 ........... SUCCESS [8:15.210s]
>>>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Jetty8  SUCCESS
[1:57:17.311s]
>>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Jetty8 ......... SUCCESS [1:02:47.374s]
>>>>> [INFO] Geronimo Assemblies :: Minimal + Tomcat7 .......... SUCCESS [8:08.838s]
>>>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Tomcat7  SUCCESS
>>>>> [1:59:16.619s]
>>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS [1:08:13.854s]
>>>>> 
>>>>> That is ~1.5 hours for the "Java EE 6 Web Profile + Jetty8" and "Java
EE 6 Web
>>>>> Profile + Tomcat7" assemblies.
>>>> 
>>>> Here are my build times. This is running without tests. Which would affect
the overall build time. Assembly times would be unaffected -- we don't run any tests when
building assemblies.
>>>> 
>>>> [INFO] Geronimo Assemblies ............................... SUCCESS [0.163s]
>>>> [INFO] Geronimo Assemblies :: Minimal + Jetty8 ........... SUCCESS [42.253s]
>>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Jetty8  SUCCESS [1:04.295s]
>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Jetty8 ......... SUCCESS [1:33.097s]
>>>> [INFO] Geronimo Assemblies :: Minimal + Tomcat7 .......... SUCCESS [41.587s]
>>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Tomcat7  SUCCESS [1:14.409s]
>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS [1:46.333s]
>>>> [INFO] ------------------------------------------------------------------------
>>>> [INFO] BUILD SUCCESS
>>>> [INFO] ------------------------------------------------------------------------
>>>> [INFO] Total time: 28:15.199s
>>>> [INFO] Finished at: Wed Jun 22 12:38:10 EDT 2011
>>>> [INFO] Final Memory: 633M/1011M
>>>> [INFO] ------------------------------------------------------------------------
>>> 
>>> Yours is about 1/3 faster than mine.
>> 
>> I think you've missed the metric. My times are an order of magnitude (hours vs minutes)
better than yours. 
>> 
>>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS [1:08:13.854s]
>> 
>> and
>> 
>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS [1:46.333s]
>> 
>> That's 68 minutes vs 1 minute 46 seconds. My *total* build time was 28 minutes.
> 
> Oh. ya. Let me rub my eyes and take another look there.... your right!
> 
>> 
>>> But you might have a faster processor too.
>>> Obviously, this part of the build is not the time consumer for me. It is nice
to
>>> know this part of the build is supposed to take several hours - that helps me
>>> narrow down the issues.
> 
> Okay - this was an incorrect statement by me.
> 
>>> 
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> For the plugins, I would assume the same, except that I notice that,
of course,
>>>>> the latest snapshot dependencies are being downloaded, and the poms in
the maven
>>>>> SNAPSHOT repositories are continually being accessed.
>>>>> 
>>>>> I realize that because trunk (G3) largely depends on 3rd-party artifact
>>>>> SNAPSHOTS that these repositories are checked during every build, and
new
>>>>> SNAPSHOT libraries are always being downloaded (as is the nature of snapshot
>>>>> releases).
>>>> 
>>>> Correct. A local maven proxy (e.g. Nexus) can reduce the overhead.
>>> 
>>> So it is worth noting as it already is now.
>>> 
>>>> 
>>>>> 
>>>>> As it is not a simple process for a user to setup a maven repo proxy,
and could
>>>>> be a deterrent to someone getting started if it were necessary to install
a
>>>>> proxy, I am looking at other methods and tips that might increase the
speed of
>>>>> building G3.
>>>> 
>>>> I've tried running with a local maven proxy. However, I've found it to be
too much trouble to maintain/configure.
>>> 
>>> Agreed. At least it is too much trouble if you are not continuously building
the
>>> entire trunk every day on the larger scale. So too much to ask a G3 beginner
to do.
>> 
>> Too much to ask of a new maven user -- I agree.
>> 
>>> 
>>>> 
>>>>> 
>>>>> Is this feasible? Or would we just say, that because of the nature of
SNAPSHOT
>>>>> builds and their dependencies on 3rd-party SNAPSHOT artifacts, there
is no real
>>>>> means to improving build speed unless a proxy is utilized?
>>>>> If the later is true, I would like to add this note in the wiki's build
>>>>> documentation. And also warn those who choose to build without a proxy
may
>>>>> experience a long build time.
>>>> 
>>>> While I'm thinking about it -- what is your network environment? The focus
on proxies imply that your slow build times are predominantly  network issue. Are you certain
that is the case?
>>> 
>>> We have about ~14Mb/s speed to the Internet. Our backplane is Gigabit Ethernet.
>>> We are connected to a major Internet backbone provider.
>>> I have been wondering if the problem is responsiveness of the maven repo sites.
>>> I have been seeing on occasion a refusal to download an artifact from the site,
>>> where as I was able to download it within a browser from the same URL.
>>> So I wonder if the repo sites are either metering the usage on a per user or
>>> session basis.
>> 
>> I'm building on my home broadband connection. Probably similar speeds. I think we
should identify the source of your extremely slow build time. 
>> 
>> If remote repository lookups are the cause of your problems, 'mvn -o clean install'
should result in a much faster build time. However, I suspect that it won't make much difference
at all...
> 
> It seems a lot of "pauses" in the build process (while using the
> '-DskipTests=true' flag) is at the end of building each plugin, and that looks
> like this (example plugin build output)...
> (Note: almost every plugin gives the log4j errors)
> -
> [INFO] ------------------------------------------------------------------------
> [INFO] Building Geronimo Plugins, Corba :: Deployer 3.0-SNAPSHOT
> [INFO] ------------------------------------------------------------------------
> [INFO]
> [INFO] --- maven-clean-plugin:2.3:clean (default-clean) @ openejb-corba-deployer ---
> [INFO]
> [INFO] --- genesis-maven-plugin:2.0:validate-configuration (default) @
> openejb-corba-deployer ---
> [INFO]
> [INFO] --- geronimo-property-plugin:3.0-SNAPSHOT:set-property (set-property) @
> openejb-corba-deployer ---
> [INFO]
> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (default) @
> openejb-corba-deployer ---
> [INFO]
> [INFO] --- maven-remote-resources-plugin:1.0:process (default) @
> openejb-corba-deployer ---
> 
> !!!!! somewhat of a pause here, can be up to 60 seconds sometimes !!!!!
> 
> [WARNING] org.apache.velocity.runtime.exception.ReferenceException: reference :
> template = META-INF/DEPENDENCIES.vm [line 40,column 14] : $license.name is not a
> valid reference.
> [WARNING] org.apache.velocity.runtime.exception.ReferenceException: reference :
> template = META-INF/DEPENDENCIES.vm [line 40,column 14] : $license.name is not a
> valid reference.
> [INFO]
> [INFO] --- maven-resources-plugin:2.3:resources (default-resources) @
> openejb-corba-deployer ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] skip non existing resourceDirectory
> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/src/main/resources
> [INFO] skip non existing resourceDirectory
> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/src/main/filtered-resources
> [INFO] Copying 3 resources
> 
> !!!!! for the plugins that use the maven-compiler-plugin:2.0.2:compile, of
> course this will take time as it is compiling !!!!!
> !!!!! However, the CPU usage only rises for like the first quarter of this wait
> time. !!!!!
> 
> [INFO]
> [INFO] --- car-maven-plugin:3.0-SNAPSHOT:validate-configuration
> (default-validate-configuration) @ openejb-corba-deployer ---
> [INFO]
> [INFO] --- car-maven-plugin:3.0-SNAPSHOT:prepare-plan (default-prepare-plan) @
> openejb-corba-deployer ---
> [INFO] Generated:
> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/work/plan.xml
> [INFO]
> [INFO] --- car-maven-plugin:3.0-SNAPSHOT:verify-no-dependency-change
> (default-verify-no-dependency-change) @ openejb-corba-deployer ---
> [INFO]
> [INFO] --- car-maven-plugin:3.0-SNAPSHOT:prepare-metadata
> (default-prepare-metadata) @ openejb-corba-deployer ---
> [INFO]
> [INFO] --- car-maven-plugin:3.0-SNAPSHOT:package (default-package) @
> openejb-corba-deployer ---
> [INFO] Packaging module configuration:
> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/work/plan.xml
> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
> : Enabling SLF4J API support.
> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
> : Enabling Jakarta Commons Logging API support.
> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
> : Enabling Log4J API support.
> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
> : Enabling Avalon Logger API support.
> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
> : Enabling JULI Logger API support.
> log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to a
> "org.apache.log4j.Appender" variable.
> log4j:ERROR The class "org.apache.log4j.Appender" was loaded by
> log4j:ERROR [25.0] whereas object of type
> log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by
> [ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340,
> parent: sun.misc.Launcher$AppClassLoader@11b86e7]].
> log4j:ERROR Could not instantiate appender named "CONSOLE".
> log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to a
> "org.apache.log4j.Appender" variable.
> log4j:ERROR The class "org.apache.log4j.Appender" was loaded by
> log4j:ERROR [25.0] whereas object of type
> log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by
> [ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340,
> parent: sun.misc.Launcher$AppClassLoader@11b86e7]].
> log4j:ERROR Could not instantiate appender named "A1".
> log4j:WARN No appenders could be found for logger
> (org.apache.geronimo.specs.geronimo-osgi-registry).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to a
> "org.apache.log4j.Appender" variable.
> log4j:ERROR The class "org.apache.log4j.Appender" was loaded by
> log4j:ERROR [25.0] whereas object of type
> log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by
> [ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340,
> parent: sun.misc.Launcher$AppClassLoader@11b86e7]].
> log4j:ERROR Could not instantiate appender named "A1".
> 
> !!!!! always a significant pause here !!!!!
> !!!!! from a few minutes to several minutes !!!!!
> 
> [INFO] Started deployer:
> org.apache.geronimo.framework/geronimo-gbean-deployer/3.0-SNAPSHOT/car
> 
> !!!!! usually a significant pause here !!!!!
> !!!!! from less than a minute to a few minutes !!!!!
> 
> [INFO]
> [INFO] --- car-maven-plugin:3.0-SNAPSHOT:archive-car (default-archive-car) @
> openejb-corba-deployer ---
> 
> !!!!! seemingly always a significant pause here !!!!!
> !!!!! from a few minutes to 5-to-10 minutes !!!!!
> 
> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
> : Disabling SLF4J API support.
> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
> : Disabling Jakarta Commons Logging API support.
> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
> : Disabling Log4J API support.
> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
> : Disabling Avalon Logger API support.
> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
> : Disabling JULI Logger API support.
> [INFO] Expanding:
> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.car
> into /tmp/archived-file-set.516894452.tmp
> [INFO] Building jar:
> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car
> [INFO] Building jar:
> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car
> [INFO]
> [INFO] --- geronimo-osgi-plugin:3.0-SNAPSHOT:verify-manifest (verify-manifest) @
> openejb-corba-deployer ---
> [INFO]
> [INFO] --- ianal-maven-plugin:1.0-alpha-1:verify-legal-files (default) @
> openejb-corba-deployer ---
> [INFO] Checking legal files in: openejb-corba-deployer-3.0-SNAPSHOT.car
> [INFO]
> [INFO] --- maven-install-plugin:2.2:install (default-install) @
> openejb-corba-deployer ---
> [INFO] Installing
> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car
> to
> /home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.car
> [INFO] Installing
> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/pom.xml
> to
> /home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.pom
> [INFO] Installing
> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/resources/META-INF/geronimo-plugin.xml
> to
> /home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.plugin-metadata
> [INFO]
> [INFO] --- car-maven-plugin:3.0-SNAPSHOT:update-pluginlist
> (default-update-pluginlist) @ openejb-corba-deployer ---
> [INFO]
> [INFO] ------------------------------------------------------------------------
> [INFO] Building Geronimo Plugins, Corba :: Client Yoko 3.0-SNAPSHOT
> [INFO] ------------------------------------------------------------------------
> ... etc ...
> -
> 
>> 
>>> 
>>> Thanks for the excellent tips!
>>> I'll try some things out, and look at how to reflect the tips in the wiki
>>> without degrading any affirmation of Best Practices (i.e. running the test cases
>>> is good).
>> 
>> No problem. Many thanks for digging into this. Definitely an area we tend to overlook...
>> 
>> --kevan
>> 


Mime
View raw message