geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell E Glaue <rgl...@cait.org>
Subject Re: Speeding up G3.0 build time
Date Wed, 22 Jun 2011 20:32:46 GMT
On 06/22/2011 01:54 PM, Kevan Miller wrote:
> 
> On Jun 22, 2011, at 1:27 PM, Russell E Glaue wrote:
> 
>> On 06/22/2011 12:02 PM, Kevan Miller wrote:
>>>
>>> On Jun 22, 2011, at 12:09 PM, Russell E Glaue wrote:
> 
> <snip>
> 
>>>
>>>>
>>>> I am using a very basic build procedure.
>>>> 1. checkout G3 trunk
>>>> 2. JAVA_HOME="/path/to/jdk/1.6.0_22"
>>>> 3. download and install a fresh maven 2.2.1
>>>> 4. MAVEN_OPTS="-Xmx968m -XX:MaxPermSize=500m -XX:ReservedCodeCacheSize=64m"
>>>
>>> My settings are similar. I don't specify '-XX:ReservedCodeCacheSize=64m'.
>>>
>>> I run with the following. I know there are others running with larger max heaps.
I haven't :
>>>
>>> MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError"

Without the "-XX:ReservedCodeCacheSize=64m" option in MAVEN_OPTS I get this
OutOfMemory error:
-
[INFO] ------------------------------------------------------------------------
[INFO] Building Geronimo Build Support :: Plugin 3.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
...<snip>...
[INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) @
buildsupport-maven-plugin ---
[INFO] Compiling 3 source files to
/data/geronimo/asf-geronimo-server-trunk/framework/buildsupport/buildsupport-maven-plugin/target/classes
[INFO]
[INFO] --- gmaven-plugin:1.2:compile (default) @ buildsupport-maven-plugin ---
---------------------------------------------------
constituent[0]: file:/usr/local/apache-maven-3.0.3/lib/maven-embedder-3.0.3.jar
...<snip>...
constituent[30]: file:/usr/local/apache-maven-3.0.3/lib/maven-compat-3.0.3.jar
---------------------------------------------------
Exception in thread "main" java.lang.VirtualMachineError: out of space in
CodeCache for adapters
        at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:212)
        at
org.codehaus.groovy.ast.builder.AstBuilderTransformation.visit(AstBuilderTransformation.groovy:62)
        at
org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:268)
        at
org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:799)
        at
org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:464)
        at
org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:443)
        at
org.codehaus.gmaven.runtime.v1_6.ClassCompilerFeature$ClassCompilerImpl.compile(ClassCompilerFeature.java:159)
        at
org.codehaus.gmaven.plugin.compile.AbstractCompileMojo.compile(AbstractCompileMojo.java:200)
        at
org.codehaus.gmaven.plugin.compile.AbstractCompileMojo.process(AbstractCompileMojo.java:164)
        at
org.codehaus.gmaven.plugin.ComponentMojoSupport.doExecute(ComponentMojoSupport.java:60)
        at org.codehaus.gmaven.plugin.MojoSupport.execute(MojoSupport.java:69)
        at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
        at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
        at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
-

With the "-XX:ReservedCodeCacheSize=64m" flag, I do not get this error.


>>
>> The server I am build on has 1024MB of memory total, and has a dual dual-core
>> Intel(R) XEON(TM) CPU 2.00GHz with 512KB cache.
>>
>> I have tried many JVM configurations, and the one like mine and yours is the
>> best one I have found. For me, a JVM configuration with less than 400m for
>> MaxPermSize always ended with an OutOfMemory error on PermGen space.
>>
>> The Geronimo dev wiki suggests 200m for MaxPermSize, which never worked for me
>> for trunk. I always assumed it was because of all the snapshot downloading the
>> build process has to do. Even after the 2nd or 3rd build, 200m was never enough.
> 
> Right. So, MaxPermSize=200m is definitely out-of-date. There is/may be some bugs that
are increasing this value. However, not really worth our while to pursue at the moment. I
don't think MaxPermSize=500m is too prohibitive. More important things to do, ATM. 
> 
> That said -- I'm building on a machine with 8 gigs of real memory. So, 1024MB of real
memory seems very small to me. And I suspect that this is the source of your problem... A
max heap of 1024MB and MaxPermSize of 512m means your system may be spending a lot of time
swapping virtual memory. 

So wrote what are your JVM Options set at:
 MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError"
Mine are:
 MAVEN_OPTS="-Xmx968m -XX:MaxPermSize=500m -XX:ReservedCodeCacheSize=64m"

The machine I am using has 1024MB of real memory and is dedicated to building
geronimo (and it is not a virtual server). I set the heap max to 968m to keep
the memory usage from peaking at 100% of system capacity.

But there is a chance you may be right about the swaping. the machine runs at
100% real memory usage (expected) and about ~50% swap.


I thinking recommending a minimum amount of real memory a machine should have to
adequately compile G3 in a reasonable amount of time should be suggested.
What would you say the minimum recommendation should be? For Windows and Linux?
2GB for Windows and 1.5GB for Linux, assuming the machine is dedicated to compiling?

> 
>>
>>>
>>>> 5. mvn clean install
>>>> 6. 13 to 39 hours later == BUILD SUCCESS
>>>>
>>>>
>>>> From observing the build process, it would seem the most used (seemingly
paused)
>>>> time is right before the build process moves on to the next plugin or assembly.
>>>>
>>>> This "seemingly paused time" after the assembly I assume is the packaging
>>>> process, because the resulting output is a huge collection.
>>>>
>>>> [INFO] Geronimo Assemblies ............................... SUCCESS [3.995s]
>>>> [INFO] Geronimo Assemblies :: Minimal + Jetty8 ........... SUCCESS [8:15.210s]
>>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Jetty8  SUCCESS [1:57:17.311s]
>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Jetty8 ......... SUCCESS [1:02:47.374s]
>>>> [INFO] Geronimo Assemblies :: Minimal + Tomcat7 .......... SUCCESS [8:08.838s]
>>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Tomcat7  SUCCESS
>>>> [1:59:16.619s]
>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS [1:08:13.854s]
>>>>
>>>> That is ~1.5 hours for the "Java EE 6 Web Profile + Jetty8" and "Java EE
6 Web
>>>> Profile + Tomcat7" assemblies.
>>>
>>> Here are my build times. This is running without tests. Which would affect the
overall build time. Assembly times would be unaffected -- we don't run any tests when building
assemblies.
>>>
>>> [INFO] Geronimo Assemblies ............................... SUCCESS [0.163s]
>>> [INFO] Geronimo Assemblies :: Minimal + Jetty8 ........... SUCCESS [42.253s]
>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Jetty8  SUCCESS [1:04.295s]
>>> [INFO] Geronimo Assemblies :: Java EE 6 + Jetty8 ......... SUCCESS [1:33.097s]
>>> [INFO] Geronimo Assemblies :: Minimal + Tomcat7 .......... SUCCESS [41.587s]
>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Tomcat7  SUCCESS [1:14.409s]
>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS [1:46.333s]
>>> [INFO] ------------------------------------------------------------------------
>>> [INFO] BUILD SUCCESS
>>> [INFO] ------------------------------------------------------------------------
>>> [INFO] Total time: 28:15.199s
>>> [INFO] Finished at: Wed Jun 22 12:38:10 EDT 2011
>>> [INFO] Final Memory: 633M/1011M
>>> [INFO] ------------------------------------------------------------------------
>>
>> Yours is about 1/3 faster than mine.
> 
> I think you've missed the metric. My times are an order of magnitude (hours vs minutes)
better than yours. 
> 
>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS [1:08:13.854s]
> 
> and
> 
>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS [1:46.333s]
> 
> That's 68 minutes vs 1 minute 46 seconds. My *total* build time was 28 minutes.

Oh. ya. Let me rub my eyes and take another look there.... your right!

> 
>> But you might have a faster processor too.
>> Obviously, this part of the build is not the time consumer for me. It is nice to
>> know this part of the build is supposed to take several hours - that helps me
>> narrow down the issues.

Okay - this was an incorrect statement by me.

>>
>>>
>>>
>>>>
>>>>
>>>> For the plugins, I would assume the same, except that I notice that, of course,
>>>> the latest snapshot dependencies are being downloaded, and the poms in the
maven
>>>> SNAPSHOT repositories are continually being accessed.
>>>>
>>>> I realize that because trunk (G3) largely depends on 3rd-party artifact
>>>> SNAPSHOTS that these repositories are checked during every build, and new
>>>> SNAPSHOT libraries are always being downloaded (as is the nature of snapshot
>>>> releases).
>>>
>>> Correct. A local maven proxy (e.g. Nexus) can reduce the overhead.
>>
>> So it is worth noting as it already is now.
>>
>>>
>>>>
>>>> As it is not a simple process for a user to setup a maven repo proxy, and
could
>>>> be a deterrent to someone getting started if it were necessary to install
a
>>>> proxy, I am looking at other methods and tips that might increase the speed
of
>>>> building G3.
>>>
>>> I've tried running with a local maven proxy. However, I've found it to be too
much trouble to maintain/configure.
>>
>> Agreed. At least it is too much trouble if you are not continuously building the
>> entire trunk every day on the larger scale. So too much to ask a G3 beginner to do.
> 
> Too much to ask of a new maven user -- I agree.
> 
>>
>>>
>>>>
>>>> Is this feasible? Or would we just say, that because of the nature of SNAPSHOT
>>>> builds and their dependencies on 3rd-party SNAPSHOT artifacts, there is no
real
>>>> means to improving build speed unless a proxy is utilized?
>>>> If the later is true, I would like to add this note in the wiki's build
>>>> documentation. And also warn those who choose to build without a proxy may
>>>> experience a long build time.
>>>
>>> While I'm thinking about it -- what is your network environment? The focus on
proxies imply that your slow build times are predominantly  network issue. Are you certain
that is the case?
>>
>> We have about ~14Mb/s speed to the Internet. Our backplane is Gigabit Ethernet.
>> We are connected to a major Internet backbone provider.
>> I have been wondering if the problem is responsiveness of the maven repo sites.
>> I have been seeing on occasion a refusal to download an artifact from the site,
>> where as I was able to download it within a browser from the same URL.
>> So I wonder if the repo sites are either metering the usage on a per user or
>> session basis.
> 
> I'm building on my home broadband connection. Probably similar speeds. I think we should
identify the source of your extremely slow build time. 
> 
> If remote repository lookups are the cause of your problems, 'mvn -o clean install' should
result in a much faster build time. However, I suspect that it won't make much difference
at all...

It seems a lot of "pauses" in the build process (while using the
'-DskipTests=true' flag) is at the end of building each plugin, and that looks
like this (example plugin build output)...
(Note: almost every plugin gives the log4j errors)
-
[INFO] ------------------------------------------------------------------------
[INFO] Building Geronimo Plugins, Corba :: Deployer 3.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.3:clean (default-clean) @ openejb-corba-deployer ---
[INFO]
[INFO] --- genesis-maven-plugin:2.0:validate-configuration (default) @
openejb-corba-deployer ---
[INFO]
[INFO] --- geronimo-property-plugin:3.0-SNAPSHOT:set-property (set-property) @
openejb-corba-deployer ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (default) @
openejb-corba-deployer ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.0:process (default) @
openejb-corba-deployer ---

!!!!! somewhat of a pause here, can be up to 60 seconds sometimes !!!!!

[WARNING] org.apache.velocity.runtime.exception.ReferenceException: reference :
template = META-INF/DEPENDENCIES.vm [line 40,column 14] : $license.name is not a
valid reference.
[WARNING] org.apache.velocity.runtime.exception.ReferenceException: reference :
template = META-INF/DEPENDENCIES.vm [line 40,column 14] : $license.name is not a
valid reference.
[INFO]
[INFO] --- maven-resources-plugin:2.3:resources (default-resources) @
openejb-corba-deployer ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/src/main/resources
[INFO] skip non existing resourceDirectory
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/src/main/filtered-resources
[INFO] Copying 3 resources

!!!!! for the plugins that use the maven-compiler-plugin:2.0.2:compile, of
course this will take time as it is compiling !!!!!
!!!!! However, the CPU usage only rises for like the first quarter of this wait
time. !!!!!

[INFO]
[INFO] --- car-maven-plugin:3.0-SNAPSHOT:validate-configuration
(default-validate-configuration) @ openejb-corba-deployer ---
[INFO]
[INFO] --- car-maven-plugin:3.0-SNAPSHOT:prepare-plan (default-prepare-plan) @
openejb-corba-deployer ---
[INFO] Generated:
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/work/plan.xml
[INFO]
[INFO] --- car-maven-plugin:3.0-SNAPSHOT:verify-no-dependency-change
(default-verify-no-dependency-change) @ openejb-corba-deployer ---
[INFO]
[INFO] --- car-maven-plugin:3.0-SNAPSHOT:prepare-metadata
(default-prepare-metadata) @ openejb-corba-deployer ---
[INFO]
[INFO] --- car-maven-plugin:3.0-SNAPSHOT:package (default-package) @
openejb-corba-deployer ---
[INFO] Packaging module configuration:
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/work/plan.xml
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling SLF4J API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling Jakarta Commons Logging API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling Log4J API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling Avalon Logger API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling JULI Logger API support.
log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to a
"org.apache.log4j.Appender" variable.
log4j:ERROR The class "org.apache.log4j.Appender" was loaded by
log4j:ERROR [25.0] whereas object of type
log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by
[ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340,
parent: sun.misc.Launcher$AppClassLoader@11b86e7]].
log4j:ERROR Could not instantiate appender named "CONSOLE".
log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to a
"org.apache.log4j.Appender" variable.
log4j:ERROR The class "org.apache.log4j.Appender" was loaded by
log4j:ERROR [25.0] whereas object of type
log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by
[ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340,
parent: sun.misc.Launcher$AppClassLoader@11b86e7]].
log4j:ERROR Could not instantiate appender named "A1".
log4j:WARN No appenders could be found for logger
(org.apache.geronimo.specs.geronimo-osgi-registry).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to a
"org.apache.log4j.Appender" variable.
log4j:ERROR The class "org.apache.log4j.Appender" was loaded by
log4j:ERROR [25.0] whereas object of type
log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by
[ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340,
parent: sun.misc.Launcher$AppClassLoader@11b86e7]].
log4j:ERROR Could not instantiate appender named "A1".

!!!!! always a significant pause here !!!!!
!!!!! from a few minutes to several minutes !!!!!

[INFO] Started deployer:
org.apache.geronimo.framework/geronimo-gbean-deployer/3.0-SNAPSHOT/car

!!!!! usually a significant pause here !!!!!
!!!!! from less than a minute to a few minutes !!!!!

[INFO]
[INFO] --- car-maven-plugin:3.0-SNAPSHOT:archive-car (default-archive-car) @
openejb-corba-deployer ---

!!!!! seemingly always a significant pause here !!!!!
!!!!! from a few minutes to 5-to-10 minutes !!!!!

org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Disabling SLF4J API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Disabling Jakarta Commons Logging API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Disabling Log4J API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Disabling Avalon Logger API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Disabling JULI Logger API support.
[INFO] Expanding:
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.car
into /tmp/archived-file-set.516894452.tmp
[INFO] Building jar:
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car
[INFO] Building jar:
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car
[INFO]
[INFO] --- geronimo-osgi-plugin:3.0-SNAPSHOT:verify-manifest (verify-manifest) @
openejb-corba-deployer ---
[INFO]
[INFO] --- ianal-maven-plugin:1.0-alpha-1:verify-legal-files (default) @
openejb-corba-deployer ---
[INFO] Checking legal files in: openejb-corba-deployer-3.0-SNAPSHOT.car
[INFO]
[INFO] --- maven-install-plugin:2.2:install (default-install) @
openejb-corba-deployer ---
[INFO] Installing
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car
to
/home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.car
[INFO] Installing
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/pom.xml
to
/home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.pom
[INFO] Installing
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/resources/META-INF/geronimo-plugin.xml
to
/home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.plugin-metadata
[INFO]
[INFO] --- car-maven-plugin:3.0-SNAPSHOT:update-pluginlist
(default-update-pluginlist) @ openejb-corba-deployer ---
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Geronimo Plugins, Corba :: Client Yoko 3.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
... etc ...
-

> 
>>
>> Thanks for the excellent tips!
>> I'll try some things out, and look at how to reflect the tips in the wiki
>> without degrading any affirmation of Best Practices (i.e. running the test cases
>> is good).
> 
> No problem. Many thanks for digging into this. Definitely an area we tend to overlook...
> 
> --kevan
> 

Mime
View raw message