hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daryn Sharp <da...@yahoo-inc.com>
Subject Re: .23 compile times
Date Thu, 16 Feb 2012 22:13:51 GMT
Wow, nice!  How long is a rebuild with no changes?

I'd be delighted if there was a dev profile that just did the bare minimum to produce a working
env.

Daryn


On Feb 16, 2012, at 3:25 PM, Alejandro Abdelnur wrote:

> I've just done some tweaks in the POMs and I'm cutting the 8.30min full
> dist build down to 3.45min. I'm removing all javadoc stuff from the dist
> profile.
> 
> I think we should introduce a developer profile that does this.
> 
> Also, the 'mvn install -DskipTests', to get the JARs in your m2 cache is
> taking 2mins. But we are building source JARs, the developer profile could
> skip the generation of the source JARs, this would cut several seconds.
> 
> Finally, note that to just build all JARs 'mvn clean test -DskipTests' it
> is taking 1.55mins.
> 
> If folk agree I'll open a JIRA to optmize the build based on these findings.
> 
> Thxs.
> 
> Alejandro
> 
> On Wed, Feb 15, 2012 at 1:02 PM, Daryn Sharp <daryn@yahoo-inc.com> wrote:
> 
>> Hi Sriram,
>> 
>> You may want to try adding your source tree to Spotlight's privacy list,
>> otherwise it's busy trying to index files as fast as the build is writing
>> them.  If you are (un)fortunate enough to be running anti-virus sw, then it
>> too is scurrying along behind your build checking every file.  That's
>> enough to turn a normally silent MBP into a hair dryer.
>> 
>> If you don't want to omit the entire project, because perhaps you want to
>> use mdfind, then you can rename dirs to end with ".noindex".  The
>> trick/hack is you can symlink the original dir name to the renamed
>> dir.noindex.
>> 
>> But yes, I think the build is way to slow to be productive when used as
>> intended.  The fact that people have to use custom tricks to build & test
>> in a "reasonable" amount of time is a symptom of a problem...
>> 
>> Daryn
>> 
>> 
>> On Feb 15, 2012, at 12:47 PM, Todd Lipcon wrote:
>> 
>>> Hi Sriram,
>>> 
>>> I also do -Dmaven.skip.javadoc=true usually and that cuts out a couple
>>> minutes. Unless you need a tarball, you can also drop -Ptar.
>>> 
>>> When I'm just making quick changes confined to the java code, I
>>> usually just use "mvn -Dskiptests install" from within the hdfs
>>> project, then manually cp the resulting
>>> target/hadoop-hdfs-0.24.0-SNAPSHOT.jar into my install
>>> dir/share/hadoop/hdfs. Much faster than the full rebuild.
>>> 
>>> -Todd
>>> 
>>> On Wed, Feb 15, 2012 at 8:52 AM, Harsh J <harsh@cloudera.com> wrote:
>>>> Sriram,
>>>> 
>>>> My package command didn't take as long with the same command -- where
>>>> do you see it getting stuck at most of the times? Which particular
>>>> target (right term?) seems to take the highest time to complete?
>>>> 
>>>> On Wed, Feb 15, 2012 at 12:31 AM, Sriram Rao <sriramsrao@gmail.com>
>> wrote:
>>>>> Folks,
>>>>> 
>>>>> The build takes forever:
>>>>> 
>>>>> mvn package -Pdist -DskipTests -Dtar
>>>>> 
>>>>> ....
>>>>> 
>>>>> [INFO]
>>>>> 
>> ------------------------------------------------------------------------
>>>>> [INFO] BUILD SUCCESS
>>>>> [INFO]
>>>>> 
>> ------------------------------------------------------------------------
>>>>> [INFO] Total time: 6:25.399s
>>>>> [INFO] Finished at: Tue Feb 14 10:54:18 PST 2012
>>>>> [INFO] Final Memory: 56M/123M
>>>>> [INFO]
>>>>> 
>> ------------------------------------------------------------------------
>>>>> 
>>>>> This is on a mac with 8GB RAM.  (The above case involved 0-lines of
>> code
>>>>> change; just making sure I had everything built!).
>>>>> 
>>>>> Is there a faster way to get the thing to build?
>>>>> 
>>>>> Sriram
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Harsh J
>>>> Customer Ops. Engineer
>>>> Cloudera | http://tiny.cloudera.com/about
>>> 
>>> 
>>> 
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>> 
>> 


Mime
View raw message