drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aditya <...@apache.org>
Subject Improving incremental build performance
Date Sun, 16 Nov 2014 01:17:08 GMT
Devs,

Following is an indicative build duration of an incremental build on my dev
machine.

[INFO] --------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Drill Root POM ......................... SUCCESS [  4.779 s]
[INFO] Drill Protocol ................................ SUCCESS [  7.741 s]
[INFO] Common (Logical Plan, Base expressions) ....... SUCCESS [  6.569 s]
[INFO] contrib/Parent Pom ............................ SUCCESS [  0.721 s]
[INFO] contrib/data/Parent Pom ....................... SUCCESS [  0.363 s]
[INFO] contrib/data/tpch-sample-data ................. SUCCESS [  2.866 s]
[INFO] exec/Parent Pom ............................... SUCCESS [  0.775 s]
[INFO] exec/Java Execution Engine .................... SUCCESS [02:11 min]
[INFO] contrib/hive-storage-plugin/Parent Pom ........ SUCCESS [  0.584 s]
[INFO] contrib/hive-storage-plugin/hive-exec-shaded .. SUCCESS [ 41.679 s]
[INFO] contrib/hive-storage-plugin/core .............. SUCCESS [  6.131 s]
[INFO] exec/JDBC Driver using dependencies ........... SUCCESS [  4.608 s]
[INFO] JDBC JAR with all dependencies ................ SUCCESS [01:12 min]
[INFO] exec/Drill expression interpreter ............. SUCCESS [ 19.640 s]
[INFO] contrib/mongo-storage-plugin .................. SUCCESS [  3.864 s]
[INFO] contrib/hbase-storage-plugin .................. SUCCESS [  5.021 s]
[INFO] Packaging and Distribution Assembly ........... SUCCESS [  9.835 s]
[INFO] contrib/sqlline ............................... SUCCESS [  0.900 s]
[INFO] --------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] --------------------------------------------------------------------
[INFO] Total time: 05:20 min
[INFO] Finished at: 2014-11-15T16:33:07-08:00
[INFO] Final Memory: 135M/1357M
[INFO] --------------------------------------------------------------------

A cursory look at the build durations suggest possible improvement in 4
modules, viz.,
"Java Execution Engine", "hive-exec-shaded", "jdbc-all" and
"expression-interpreter".

For "Java Execution Engine", we need to fix the FreeMarker Pre Processor
maven plugin
and maven-compiler-plugin to not regenerate/recompile stuff with no
modification
since last build. I already have a patch[1] for maven-compiler-plugin,
which removes 40
seconds from an incremental build and am looking at the FMPP plugin[2].
With both the
changes in place, I expect the incremental build time for this module to
drop from 2:11
minutes to order of few seconds.

Building "hive-exec-shaded" is another possible optimization. It appears
that the module
does not depend on anything in the Drill source code and can be built
separately, on need
basis, and published to a maven repository[3]. This will shave off another
40 seconds.

Have not looked at the last two "jdbc-all" and "expression-interpreter" for
any possible
improvement yet but there might be some possibilities there too if someone
is
interested/frustrated enough :)

I have converted the JIRA[4] created by Daniel into and umbrella task and
added
sub-tasks to address these issues.

aditya...

[1] https://issues.apache.org/jira/browse/DRILL-1720
[2] https://issues.apache.org/jira/browse/DRILL-1721
[3] https://issues.apache.org/jira/browse/DRILL-1722
[4] https://issues.apache.org/jira/browse/DRILL-1717

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message