hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: Build performances
Date Tue, 26 Feb 2013 13:36:59 GMT
Hi Stack,

3 tests failed and 1271 passed.

So it seems to be pretty stable with openJDK1.7.

I retried and got only 2 failed.

In both cases testDeleteEditUnknownColumnFamilyAndOrTable was on the list.

I'm retrying another time, but for small tests, I will say that open
JDK is stable enought.

JM

2013/2/25 Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
> Hi Stack,
>
> When I'm testing locally with VMs I'm using openJDK for the standalone
> mode so I don't have to install any additionnal JVM. And it's working
> for my needs. I have not faced any issue so far.
>
> I just started the entire test suite with OpenJDK1.7. I will keep you posted.
>
> JM
>
> 2013/2/25 Stack <stack@duboce.net>:
>> Does hbase work on openjdk?
>> St.Ack
>>
>>
>> On Mon, Feb 25, 2013 at 12:54 PM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org> wrote:
>>
>>> Hi Stack,
>>>
>>> Thanks for pointing to stracing.
>>>
>>> However, I fixed my issue...
>>>
>>> Initially I was running with that:
>>> Java version: 1.6.0_24, vendor: Sun Microsystems Inc.
>>> Java home: /usr/lib/jvm/java-6-openjdk-amd64/jre
>>>
>>>
>>> With "export JAVA_HOME=/usr/local/jdk1.7.0_13/":
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD SUCCESS
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 5:37.573s
>>> [INFO] Finished at: Mon Feb 25 15:50:26 EST 2013
>>> [INFO] Final Memory: 62M/3020M
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> Less  than 6 minutes. 10 times faster!!!!
>>>
>>> So I don't really know why it was so slow with the 1.6 OpenJDK, but
>>> it's now running way faster!
>>>
>>> Thanks,
>>>
>>> JM
>>>
>>> 2013/2/25 Stack <stack@duboce.net>:
>>> > Try stracing the process [1]?
>>> > St.Ack
>>> >
>>> > 1. http://www.cyberciti.biz/tips/linux-strace-command-examples.html
>>> >
>>> >
>>> > On Mon, Feb 25, 2013 at 12:09 PM, Jean-Marc Spaggiari <
>>> > jean-marc@spaggiari.org> wrote:
>>> >
>>> >> When it's stuck on "Copying", there is nothing more displayed.
>>> >>
>>> >> Before that, it's verbose.
>>> >>
>>> >> [DEBUG] Adding file-set in: /mnt/security to archive location:
>>> >> hbase-0.94.6-SNAPSHOT/security/
>>> >> [DEBUG] FileSet[conf/] dir perms: 47777 file perms: 107777
>>> >> [DEBUG] The archive base directory is 'null'
>>> >> [DEBUG] NOT reformatting any files in /mnt/conf
>>> >> [DEBUG] Adding file-set from directory: '/mnt/conf'
>>> >> assembly output directory is: 'conf/'
>>> >> [DEBUG] Adding file-set in: /mnt/conf to archive location:
>>> >> hbase-0.94.6-SNAPSHOT/conf/
>>> >> [DEBUG] FileSet[bin/] dir perms: 47777 file perms: 107777
>>> >> [DEBUG] The archive base directory is 'null'
>>> >> [DEBUG] NOT reformatting any files in /mnt/bin
>>> >> [DEBUG] Adding file-set from directory: '/mnt/bin'
>>> >> assembly output directory is: 'bin/'
>>> >> [DEBUG] Adding file-set in: /mnt/bin to archive location:
>>> >> hbase-0.94.6-SNAPSHOT/bin/
>>> >> [DEBUG] FileSet[lib/ruby/] dir perms: 47777 file perms: 107777
>>> >> [DEBUG] The archive base directory is 'null'
>>> >> [DEBUG] NOT reformatting any files in /mnt/src/main/ruby
>>> >> [DEBUG] Adding file-set from directory: '/mnt/src/main/ruby'
>>> >> assembly output directory is: 'lib/ruby/'
>>> >> [DEBUG] Adding file-set in: /mnt/src/main/ruby to archive location:
>>> >> hbase-0.94.6-SNAPSHOT/lib/ruby/
>>> >> [DEBUG] FileSet[] dir perms: 47777 file perms: 107777
>>> >> [DEBUG] The archive base directory is 'null'
>>> >> [DEBUG] NOT reformatting any files in /mnt/target
>>> >> [DEBUG] Adding file-set from directory: '/mnt/target'
>>> >> assembly output directory is: ''
>>> >> [DEBUG] Adding file-set in: /mnt/target to archive location:
>>> >> hbase-0.94.6-SNAPSHOT/
>>> >> [DEBUG] FileSet[hbase-webapps/] dir perms: 47777 file perms: 107777
>>> >> [DEBUG] The archive base directory is 'null'
>>> >> [DEBUG] NOT reformatting any files in /mnt/target/hbase-webapps
>>> >> [DEBUG] Adding file-set from directory: '/mnt/target/hbase-webapps'
>>> >> assembly output directory is: 'hbase-webapps/'
>>> >> [DEBUG] Adding file-set in: /mnt/target/hbase-webapps to archive
>>> >> location: hbase-0.94.6-SNAPSHOT/hbase-webapps/
>>> >> [DEBUG] FileSet[docs/] dir perms: 47777 file perms: 107777
>>> >> [DEBUG] The archive base directory is 'null'
>>> >> [DEBUG] NOT reformatting any files in /mnt/target/site
>>> >> [DEBUG] Adding file-set from directory: '/mnt/target/site'
>>> >> assembly output directory is: 'docs/'
>>> >> [DEBUG] Adding file-set in: /mnt/target/site to archive location:
>>> >> hbase-0.94.6-SNAPSHOT/docs/
>>> >> [DEBUG] FileSet[sbin/] dir perms: 47777 file perms: 107777
>>> >> [DEBUG] The archive base directory is 'null'
>>> >> [DEBUG] NOT reformatting any files in /mnt/src/packages
>>> >> [DEBUG] Adding file-set from directory: '/mnt/src/packages'
>>> >> assembly output directory is: 'sbin/'
>>> >> [DEBUG] Adding file-set in: /mnt/src/packages to archive location:
>>> >> hbase-0.94.6-SNAPSHOT/sbin/
>>> >> [INFO] Copying files to /mnt/target/hbase-0.94.6-SNAPSHOT
>>> >>
>>> >>
>>> >> But now it's stuck again on the last line, not giving more
>>> >> information. I killed it and I moved everything on tmpfs but it's not
>>> >> faster. I have nothing else running.
>>> >>
>>> >> How long should it normaly take?
>>> >>
>>> >> Entire log are avaiable there: http://www.spaggiari.org/log.txt
>>> >>
>>> >> JM
>>> >>
>>> >> 2013/2/25 Stack <stack@duboce.net>:
>>> >> > Following on from Andrew, does the debug show you anything?  (Copying
>>> >> > individual files?)
>>> >> > St.Ack
>>> >> >
>>> >> >
>>> >> > On Mon, Feb 25, 2013 at 11:09 AM, Jean-Marc Spaggiari <
>>> >> > jean-marc@spaggiari.org> wrote:
>>> >> >
>>> >> >> Hi Andy.
>>> >> >>
>>> >> >> With DEBUG enabled, what seems to be slow is that:
>>> >> >> [DEBUG] Adding artifact: org.apache.commons:commons-math:jar:2.1
with
>>> >> >> file:
>>> >> >>
>>> >>
>>> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
>>> >> >> to assembly location: lib/commons-math-2.1.jar.
>>> >> >> [DEBUG] Adding file:
>>> >> >>
>>> >> >>
>>> >>
>>> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
>>> >> >> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-math-2.1.jar
>>> >> >> [DEBUG] Adding artifact: commons-net:commons-net:jar:1.4.1
with file:
>>> >> >>
>>> >> >>
>>> >>
>>> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
>>> >> >> to assembly location: lib/commons-net-1.4.1.jar.
>>> >> >> [DEBUG] Adding file:
>>> >> >>
>>> >> >>
>>> >>
>>> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
>>> >> >> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-net-1.4.1.jar
>>> >> >> About 3 minutes, displaying one line every 1 to 2 seconds.
>>> >> >>
>>> >> >> Then this is taking a lot of time:
>>> >> >> [INFO] Copying files to /tmp/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
>>> >> >>
>>> >> >> It's now running for 10 minutes and seems it's doing nothing.
This is
>>> >> >> what I saw without the debug enabled. I'm building on an SSD
drive.
>>> >> >> It's only about 250MB. I can even build it in memory if required.
But
>>> >> >> I don't think it's because of the disk.
>>> >> >>
>>> >> >> JM
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> 2013/2/25 Andrew Purtell <apurtell@apache.org>:
>>> >> >> > If you run Maven with debug logging enabled, do you see
it taking
>>> ~3
>>> >> >> > seconds per dependency jar to decide if it wants to include
it in
>>> the
>>> >> >> > assembly or not?
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > On Mon, Feb 25, 2013 at 10:42 AM, Jean-Marc Spaggiari
<
>>> >> >> > jean-marc@spaggiari.org> wrote:
>>> >> >> >
>>> >> >> >> Hi,
>>> >> >> >>
>>> >> >> >> Buliding HBase locally is taking 1h for me:
>>> >> >> >> [INFO]
>>> >> >> >>
>>> >> ------------------------------------------------------------------------
>>> >> >> >> [INFO] BUILD SUCCESS
>>> >> >> >> [INFO]
>>> >> >> >>
>>> >> ------------------------------------------------------------------------
>>> >> >> >> [INFO] Total time: 59:36.659s
>>> >> >> >> [INFO] Finished at: Mon Feb 25 13:11:36 EST 2013
>>> >> >> >> [INFO] Final Memory: 64M/593M
>>> >> >> >> [INFO]
>>> >> >> >>
>>> >> ------------------------------------------------------------------------
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> But from that our, it seems to be spending most of
the time doing
>>> >> that:
>>> >> >> >> [INFO] --- maven-assembly-plugin:2.2-beta-5:single
(tarball) @
>>> hbase
>>> >> ---
>>> >> >> >> [INFO] Reading assembly descriptor: src/assembly/all.xml
>>> >> >> >> [INFO] Copying files to
>>> >> >> >> /home/jmspaggi/workspace/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
>>> >> >> >>
>>> >> >> >> My CPUs are not loaded at all (1.44/8 load5), CPUWIO
is almost 0
>>> too,
>>> >> >> >> and disks are almost not working.
>>> >> >> >>
>>> >> >> >> I have started the build with mvn clean site install
>>> >> assembly:assembly
>>> >> >> >> -DskipTests -Prelease.
>>> >> >> >>
>>> >> >> >> Is that really spending all this time copÅ·ing files?
I will move
>>> the
>>> >> >> >> build to an SSD drive and retry, but is there anyway
to reduce the
>>> >> >> >> build time?
>>> >> >> >>
>>> >> >> >> Thanks,
>>> >> >> >>
>>> >> >> >> JM
>>> >> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > --
>>> >> >> > Best regards,
>>> >> >> >
>>> >> >> >    - Andy
>>> >> >> >
>>> >> >> > Problems worthy of attack prove their worth by hitting
back. - Piet
>>> >> Hein
>>> >> >> > (via Tom White)
>>> >> >>
>>> >>
>>>

Mime
View raw message