hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Build performances
Date Mon, 25 Feb 2013 20:42:25 GMT
Try stracing the process [1]?
St.Ack

1. http://www.cyberciti.biz/tips/linux-strace-command-examples.html


On Mon, Feb 25, 2013 at 12:09 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:

> When it's stuck on "Copying", there is nothing more displayed.
>
> Before that, it's verbose.
>
> [DEBUG] Adding file-set in: /mnt/security to archive location:
> hbase-0.94.6-SNAPSHOT/security/
> [DEBUG] FileSet[conf/] dir perms: 47777 file perms: 107777
> [DEBUG] The archive base directory is 'null'
> [DEBUG] NOT reformatting any files in /mnt/conf
> [DEBUG] Adding file-set from directory: '/mnt/conf'
> assembly output directory is: 'conf/'
> [DEBUG] Adding file-set in: /mnt/conf to archive location:
> hbase-0.94.6-SNAPSHOT/conf/
> [DEBUG] FileSet[bin/] dir perms: 47777 file perms: 107777
> [DEBUG] The archive base directory is 'null'
> [DEBUG] NOT reformatting any files in /mnt/bin
> [DEBUG] Adding file-set from directory: '/mnt/bin'
> assembly output directory is: 'bin/'
> [DEBUG] Adding file-set in: /mnt/bin to archive location:
> hbase-0.94.6-SNAPSHOT/bin/
> [DEBUG] FileSet[lib/ruby/] dir perms: 47777 file perms: 107777
> [DEBUG] The archive base directory is 'null'
> [DEBUG] NOT reformatting any files in /mnt/src/main/ruby
> [DEBUG] Adding file-set from directory: '/mnt/src/main/ruby'
> assembly output directory is: 'lib/ruby/'
> [DEBUG] Adding file-set in: /mnt/src/main/ruby to archive location:
> hbase-0.94.6-SNAPSHOT/lib/ruby/
> [DEBUG] FileSet[] dir perms: 47777 file perms: 107777
> [DEBUG] The archive base directory is 'null'
> [DEBUG] NOT reformatting any files in /mnt/target
> [DEBUG] Adding file-set from directory: '/mnt/target'
> assembly output directory is: ''
> [DEBUG] Adding file-set in: /mnt/target to archive location:
> hbase-0.94.6-SNAPSHOT/
> [DEBUG] FileSet[hbase-webapps/] dir perms: 47777 file perms: 107777
> [DEBUG] The archive base directory is 'null'
> [DEBUG] NOT reformatting any files in /mnt/target/hbase-webapps
> [DEBUG] Adding file-set from directory: '/mnt/target/hbase-webapps'
> assembly output directory is: 'hbase-webapps/'
> [DEBUG] Adding file-set in: /mnt/target/hbase-webapps to archive
> location: hbase-0.94.6-SNAPSHOT/hbase-webapps/
> [DEBUG] FileSet[docs/] dir perms: 47777 file perms: 107777
> [DEBUG] The archive base directory is 'null'
> [DEBUG] NOT reformatting any files in /mnt/target/site
> [DEBUG] Adding file-set from directory: '/mnt/target/site'
> assembly output directory is: 'docs/'
> [DEBUG] Adding file-set in: /mnt/target/site to archive location:
> hbase-0.94.6-SNAPSHOT/docs/
> [DEBUG] FileSet[sbin/] dir perms: 47777 file perms: 107777
> [DEBUG] The archive base directory is 'null'
> [DEBUG] NOT reformatting any files in /mnt/src/packages
> [DEBUG] Adding file-set from directory: '/mnt/src/packages'
> assembly output directory is: 'sbin/'
> [DEBUG] Adding file-set in: /mnt/src/packages to archive location:
> hbase-0.94.6-SNAPSHOT/sbin/
> [INFO] Copying files to /mnt/target/hbase-0.94.6-SNAPSHOT
>
>
> But now it's stuck again on the last line, not giving more
> information. I killed it and I moved everything on tmpfs but it's not
> faster. I have nothing else running.
>
> How long should it normaly take?
>
> Entire log are avaiable there: http://www.spaggiari.org/log.txt
>
> JM
>
> 2013/2/25 Stack <stack@duboce.net>:
> > Following on from Andrew, does the debug show you anything?  (Copying
> > individual files?)
> > St.Ack
> >
> >
> > On Mon, Feb 25, 2013 at 11:09 AM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> Hi Andy.
> >>
> >> With DEBUG enabled, what seems to be slow is that:
> >> [DEBUG] Adding artifact: org.apache.commons:commons-math:jar:2.1 with
> >> file:
> >>
> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
> >> to assembly location: lib/commons-math-2.1.jar.
> >> [DEBUG] Adding file:
> >>
> >>
> /home/jmspaggi/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
> >> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-math-2.1.jar
> >> [DEBUG] Adding artifact: commons-net:commons-net:jar:1.4.1 with file:
> >>
> >>
> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
> >> to assembly location: lib/commons-net-1.4.1.jar.
> >> [DEBUG] Adding file:
> >>
> >>
> /home/jmspaggi/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar
> >> to archive location: hbase-0.94.6-SNAPSHOT/lib/commons-net-1.4.1.jar
> >> About 3 minutes, displaying one line every 1 to 2 seconds.
> >>
> >> Then this is taking a lot of time:
> >> [INFO] Copying files to /tmp/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
> >>
> >> It's now running for 10 minutes and seems it's doing nothing. This is
> >> what I saw without the debug enabled. I'm building on an SSD drive.
> >> It's only about 250MB. I can even build it in memory if required. But
> >> I don't think it's because of the disk.
> >>
> >> JM
> >>
> >>
> >>
> >> 2013/2/25 Andrew Purtell <apurtell@apache.org>:
> >> > If you run Maven with debug logging enabled, do you see it taking ~3
> >> > seconds per dependency jar to decide if it wants to include it in the
> >> > assembly or not?
> >> >
> >> >
> >> >
> >> > On Mon, Feb 25, 2013 at 10:42 AM, Jean-Marc Spaggiari <
> >> > jean-marc@spaggiari.org> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> Buliding HBase locally is taking 1h for me:
> >> >> [INFO]
> >> >>
> ------------------------------------------------------------------------
> >> >> [INFO] BUILD SUCCESS
> >> >> [INFO]
> >> >>
> ------------------------------------------------------------------------
> >> >> [INFO] Total time: 59:36.659s
> >> >> [INFO] Finished at: Mon Feb 25 13:11:36 EST 2013
> >> >> [INFO] Final Memory: 64M/593M
> >> >> [INFO]
> >> >>
> ------------------------------------------------------------------------
> >> >>
> >> >>
> >> >> But from that our, it seems to be spending most of the time doing
> that:
> >> >> [INFO] --- maven-assembly-plugin:2.2-beta-5:single (tarball) @ hbase
> ---
> >> >> [INFO] Reading assembly descriptor: src/assembly/all.xml
> >> >> [INFO] Copying files to
> >> >> /home/jmspaggi/workspace/hbase-0.94/target/hbase-0.94.6-SNAPSHOT
> >> >>
> >> >> My CPUs are not loaded at all (1.44/8 load5), CPUWIO is almost 0 too,
> >> >> and disks are almost not working.
> >> >>
> >> >> I have started the build with mvn clean site install
> assembly:assembly
> >> >> -DskipTests -Prelease.
> >> >>
> >> >> Is that really spending all this time copÅ·ing files? I will move the
> >> >> build to an SSD drive and retry, but is there anyway to reduce the
> >> >> build time?
> >> >>
> >> >> Thanks,
> >> >>
> >> >> JM
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> >
> >> >    - Andy
> >> >
> >> > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >> > (via Tom White)
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message