hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-1114) Speed up ivy resolution in builds with clever caching
Date Thu, 15 Oct 2009 05:26:31 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-1114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Todd Lipcon updated MAPREDUCE-1114:

    Attachment: mapreduce-1114.txt

Attaching patch which speeds up a contrib-skipping null test case run from 32 seconds to 15
seconds wallclock.

todd@todd-laptop:~/git/hadoop-mapreduce$ time ant test -Dskip.contrib=1 -Dtestcase=xxxx >

real    0m11.360s
user    0m15.897s
sys     0m1.436s
todd@todd-laptop:~/git/hadoop-mapreduce$ git reset --hard 'HEAD^'
HEAD is now at 8ac54e1 MAPREDUCE-1113. Fix mumak to not compile aspects with skip.contrib
is set
todd@todd-laptop:~/git/hadoop-mapreduce$ time ant test -Dskip.contrib=1 -Dtestcase=xxxx >

real    0m25.222s
user    0m32.870s
sys     0m1.948s

There's still a bit of room to improve here - SLF4J is missing some uptodate check which cause
an unnecessary javac pass.

The way this works is that the macro writes the resolved classpath into build/ivy/ivy-resolve-cache/<project>-<conf>
if it doesn't already exist. If that does exist, it loads that file into the appropriate properties
as if ivy had done the resolution anew.

This patch is not final quality - it outputs some debugging printouts and could be cleaned
up a bit, just wanted to see what people thought before spending the time to make it prettier.

You may wonder why this is starting as a MAPREDUCE patch and not HADOOP - no particular reason.
If people like this I will do the same for common and HDFS.

> Speed up ivy resolution in builds with clever caching
> -----------------------------------------------------
>                 Key: MAPREDUCE-1114
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1114
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: mapreduce-1114.txt
> An awful lot of time is spent in the ivy:resolve parts of the build, even when all of
the dependencies have been fetched and cached. Profiling showed this was in XML parsing. I
have a sort-of-ugly hack which speeds up incremental compiles (and more importantly "ant test")
significantly using some ant macros to cache the resolved classpaths.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message