hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Boudnik (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1114) Speed up ivy resolution in builds with clever caching
Date Thu, 10 Dec 2009 03:11:18 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788462#action_12788462

Konstantin Boudnik commented on MAPREDUCE-1114:

May be I'm barking on a wrong tree but I've tried to run a couple of commands in current MR
 % time ant ivy-resolve-common
Buildfile: build.xml
Total time: 3 seconds

real    0m4.513s
user    0m5.186s
sys     0m0.616s

I got very close result for {{% time ant ivy-resolve-tree}}

so it seems to me that resolver works pretty damn fast considering the number of artifacts
it needs to check. May be the latency is hiding somewhere else?

> Speed up ivy resolution in builds with clever caching
> -----------------------------------------------------
>                 Key: MAPREDUCE-1114
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1114
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: mapreduce-1114.txt, mapreduce-1114.txt, mapreduce-1114.txt
> An awful lot of time is spent in the ivy:resolve parts of the build, even when all of
the dependencies have been fetched and cached. Profiling showed this was in XML parsing. I
have a sort-of-ugly hack which speeds up incremental compiles (and more importantly "ant test")
significantly using some ant macros to cache the resolved classpaths.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message