Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.173 is neither permitted
 nor denied by domain of daryn@yahoo-inc.com)
From: Daryn Sharp <daryn@yahoo-inc.com>
To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
Date: Fri, 17 Feb 2012 09:03:11 -0600
Subject: Re: .23 compile times
Thread-Topic: .23 compile times
Thread-Index: AczthUT3kK3TbYfoQSefn6ZF0FbIzQ==
Message-ID: <9418257B-EFFC-489C-BD19-CB4F2EC56AAC@yahoo-inc.com>
References: 
 <CAEsa9_UnLjEPvOk3djTvE7JBFuDREubO+oTWKOnmjd=_kF16Rg@mail.gmail.com>
 <CAOcnVr1uWPdG0mtaE5xKNdLz9Yy5D-Es39nw8G-FYgiayiQ2xw@mail.gmail.com>
 <CADY20s7_tBrGrwLn4NwJ2A4K5MpXNEZ-nk2oZMaRvDXovFN4tA@mail.gmail.com>
 <22102138-E9FE-4443-B0E4-153EAA3D2F47@yahoo-inc.com>
 <CAJs-t7MwcyWLQw9sXF5Lz34KEWJ-4TDZF0TOyoAo7DsoANukfA@mail.gmail.com>
 <98E6117B-D0A1-4F57-95BC-87F504281008@yahoo-inc.com>
 <CAJs-t7Ovpb5gK-_0mSLG09evr0E9vG=jodcWFBZdYg+px4=hEw@mail.gmail.com>
In-Reply-To: 
 <CAJs-t7Ovpb5gK-_0mSLG09evr0E9vG=jodcWFBZdYg+px4=hEw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Very nice improvement!  How long is "mvn install -DskipTests -Dmaven.javado=
c.skip"?  Assuming that's fast, is there possibly a way to allow hadoop to =
run out of the checkedout tree?  Maybe through the CLASSPATH, or by subproj=
ects copying the built jars into the dist area?  The ideal would be to run =
a full maven build, followed by just a mvn build within a specific subproje=
ct.

Daryn


On Feb 16, 2012, at 6:15 PM, Alejandro Abdelnur wrote:

> A 'mvn package -DskipTests -Pdist'  (developer) with no changes takes
> 3.35mins
>=20
> The biggest offenders are:
>=20
> [INFO] Apache Hadoop Common .............................. SUCCESS [43.29=
9s]
> [INFO] Apache Hadoop HDFS ................................ SUCCESS [54.77=
6s]
> [INFO] hadoop-yarn-api ................................... SUCCESS [15.46=
6s]
> [INFO] hadoop-mapreduce .................................. SUCCESS [16.15=
4s]
> [INFO] Apache Hadoop Distributed Copy .................... SUCCESS [9.657=
s]
>=20
> Everybody else is below 5secs
>=20
> A 'mvn test -DskipTests' with no changes 1.05mins (this should go down
> making the some codegen to be smart instead just compiling again)
>=20
> The biggest offenders are:
>=20
> [INFO] Apache Hadoop Common .............................. SUCCESS [9.121=
s]
> [INFO] Apache Hadoop HDFS ................................ SUCCESS [20.88=
7s]
> [INFO] hadoop-yarn-api ................................... SUCCESS [6.664=
s]
>=20
> A few 2sec modules and the rest below 1sec
>=20
> Thanks.
>=20
> Alejandro
>=20
>=20
> On Thu, Feb 16, 2012 at 2:13 PM, Daryn Sharp <daryn@yahoo-inc.com> wrote:
>=20
>> Wow, nice!  How long is a rebuild with no changes?
>>=20
>> I'd be delighted if there was a dev profile that just did the bare minim=
um
>> to produce a working env.
>>=20
>> Daryn
>>=20
>>=20
>> On Feb 16, 2012, at 3:25 PM, Alejandro Abdelnur wrote:
>>=20
>>> I've just done some tweaks in the POMs and I'm cutting the 8.30min full
>>> dist build down to 3.45min. I'm removing all javadoc stuff from the dis=
t
>>> profile.
>>>=20
>>> I think we should introduce a developer profile that does this.
>>>=20
>>> Also, the 'mvn install -DskipTests', to get the JARs in your m2 cache i=
s
>>> taking 2mins. But we are building source JARs, the developer profile
>> could
>>> skip the generation of the source JARs, this would cut several seconds.
>>>=20
>>> Finally, note that to just build all JARs 'mvn clean test -DskipTests' =
it
>>> is taking 1.55mins.
>>>=20
>>> If folk agree I'll open a JIRA to optmize the build based on these
>> findings.
>>>=20
>>> Thxs.
>>>=20
>>> Alejandro
>>>=20
>>> On Wed, Feb 15, 2012 at 1:02 PM, Daryn Sharp <daryn@yahoo-inc.com>
>> wrote:
>>>=20
>>>> Hi Sriram,
>>>>=20
>>>> You may want to try adding your source tree to Spotlight's privacy lis=
t,
>>>> otherwise it's busy trying to index files as fast as the build is
>> writing
>>>> them.  If you are (un)fortunate enough to be running anti-virus sw,
>> then it
>>>> too is scurrying along behind your build checking every file.  That's
>>>> enough to turn a normally silent MBP into a hair dryer.
>>>>=20
>>>> If you don't want to omit the entire project, because perhaps you want
>> to
>>>> use mdfind, then you can rename dirs to end with ".noindex".  The
>>>> trick/hack is you can symlink the original dir name to the renamed
>>>> dir.noindex.
>>>>=20
>>>> But yes, I think the build is way to slow to be productive when used a=
s
>>>> intended.  The fact that people have to use custom tricks to build &
>> test
>>>> in a "reasonable" amount of time is a symptom of a problem...
>>>>=20
>>>> Daryn
>>>>=20
>>>>=20
>>>> On Feb 15, 2012, at 12:47 PM, Todd Lipcon wrote:
>>>>=20
>>>>> Hi Sriram,
>>>>>=20
>>>>> I also do -Dmaven.skip.javadoc=3Dtrue usually and that cuts out a cou=
ple
>>>>> minutes. Unless you need a tarball, you can also drop -Ptar.
>>>>>=20
>>>>> When I'm just making quick changes confined to the java code, I
>>>>> usually just use "mvn -Dskiptests install" from within the hdfs
>>>>> project, then manually cp the resulting
>>>>> target/hadoop-hdfs-0.24.0-SNAPSHOT.jar into my install
>>>>> dir/share/hadoop/hdfs. Much faster than the full rebuild.
>>>>>=20
>>>>> -Todd
>>>>>=20
>>>>> On Wed, Feb 15, 2012 at 8:52 AM, Harsh J <harsh@cloudera.com> wrote:
>>>>>> Sriram,
>>>>>>=20
>>>>>> My package command didn't take as long with the same command -- wher=
e
>>>>>> do you see it getting stuck at most of the times? Which particular
>>>>>> target (right term?) seems to take the highest time to complete?
>>>>>>=20
>>>>>> On Wed, Feb 15, 2012 at 12:31 AM, Sriram Rao <sriramsrao@gmail.com>
>>>> wrote:
>>>>>>> Folks,
>>>>>>>=20
>>>>>>> The build takes forever:
>>>>>>>=20
>>>>>>> mvn package -Pdist -DskipTests -Dtar
>>>>>>>=20
>>>>>>> ....
>>>>>>>=20
>>>>>>> [INFO]
>>>>>>>=20
>>>> ----------------------------------------------------------------------=
--
>>>>>>> [INFO] BUILD SUCCESS
>>>>>>> [INFO]
>>>>>>>=20
>>>> ----------------------------------------------------------------------=
--
>>>>>>> [INFO] Total time: 6:25.399s
>>>>>>> [INFO] Finished at: Tue Feb 14 10:54:18 PST 2012
>>>>>>> [INFO] Final Memory: 56M/123M
>>>>>>> [INFO]
>>>>>>>=20
>>>> ----------------------------------------------------------------------=
--
>>>>>>>=20
>>>>>>> This is on a mac with 8GB RAM.  (The above case involved 0-lines of
>>>> code
>>>>>>> change; just making sure I had everything built!).
>>>>>>>=20
>>>>>>> Is there a faster way to get the thing to build?
>>>>>>>=20
>>>>>>> Sriram
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>> --
>>>>>> Harsh J
>>>>>> Customer Ops. Engineer
>>>>>> Cloudera | http://tiny.cloudera.com/about
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>>=20
>>>>=20
>>=20
>>=20