hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Maven assembly speed issue (from discussion with jdcryans on IRC)
Date Tue, 16 Mar 2010 23:56:38 GMT
Here is a list of things I found in lib that we don't need to run:

-rwsrwsrwt   1 Stack  staff  1034049 Mar 16 16:45 ant-1.6.5.jar
-rwsrwsrwt   1 Stack  staff   279781 Mar 16 16:45 commons-httpclient-3.0.1.jar
-rwsrwsrwt   1 Stack  staff   706710 Mar 16 16:45 hsqldb-1.8.0.10.jar
-rwsrwsrwt   1 Stack  staff   377780 Mar 16 16:45 jets3t-0.7.1.jar
-rwsrwsrwt   1 Stack  staff    11981 Mar 16 16:45 kfs-0.3.jar
-rwsrwsrwt   1 Stack  staff   388864 Mar 16 16:45 mail-1.4.jar
-rwsrwsrwt   1 Stack  staff    65261 Mar 16 16:45 oro-2.0.8.jar
-rwsrwsrwt   1 Stack  staff    28415 Mar 16 16:45 paranamer-1.5.jar
-rwsrwsrwt   1 Stack  staff     3067 Mar 16 16:45 paranamer-ant-1.5.jar
-rwsrwsrwt   1 Stack  staff     6841 Mar 16 16:45 paranamer-generator-1.5.jar
-rwsrwsrwt   1 Stack  staff   167436 Mar 16 16:45 qdox-1.9.1.jar
-rwsrwsrwt   1 Stack  staff    23445 Mar 16 16:45 slf4j-api-1.5.8.jar
-rwsrwsrwt   1 Stack  staff     9679 Mar 16 16:45 slf4j-log4j12-1.5.8.jar
-rwsrwsrwt   1 Stack  staff     7585 Mar 16 16:45 slf4j-simple-1.5.8.jar

To add exclusions, I need to add the exclusion on the dependency that
adds in the above?  For most of the above, it looks like hadoop is the
includer.  I'd look at its pom and figure which hadoop component
included the above and then add an exclusion in our pom on that
dependency?

Thanks Paul,
St.Ack


On Tue, Mar 16, 2010 at 3:12 PM, Paul Smith <psmith@aconex.com> wrote:
> just a follow up on this one:
>>
>>
>>
>> A bit of research I came across this issue with the Maven Assembly plugin.
>>
>> http://jira.codehaus.org/browse/MASSEMBLY-424
>>
>
> I've posted some comments on this issue, but it basically comes down to a process fork
per dependency which is a bit ScalabilityYucky(tm). It will do this multiple times for a single
jar because the hbase modules are interrelated (they end up sharing a big chunk of the dependency
space).
>
> We're in a hard place here because going back to beta-1 of the plugin would require some
rework of the pom.xml to match the required syntax for that version.  For now though stack
is rightly sticking with beta-5 for now, sluggish though that is.
>
> One other way we could do to speed this up is now look at the dependency tree and start
considering setting up exclude rules for those transitive dependencies you guys just know
is not needed.    By removing a jar, we reduce at least one, probably multiple process forks.
>
> I think if someone could inspect the 'lib' directory of the tar ball generated by mvn
and say "hey, these X jars can just go" I or someone else can easily add the exclusions.
>
> thoughts?
>
> Paul

Mime
View raw message