hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14284) Shade Guava everywhere
Date Tue, 02 May 2017 00:36:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991891#comment-15991891
] 

Junping Du commented on HADOOP-14284:
-------------------------------------

bq. I'm pretty sure unit tests still dominate the build time.
I mean skip the unit test.

bq. After that, I wouldn't be surprised if it's YARN.
>From my local report for mvn install, no evidence to show YARN have any relationship with
build slowness. All shading related components are dominating the build speed:

{noformat}
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.324 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  2.295 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.542 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  1.355 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  0.111 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.150 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  2.337 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  0.662 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  4.585 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.044 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 27.720 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  2.067 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  3.101 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.075 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 21.961 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 27.508 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  0.475 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 16.516 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  1.081 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.049 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.052 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [  8.417 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 11.146 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.064 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [  3.441 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [  5.758 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  1.541 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  2.741 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  2.264 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 14.351 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  2.159 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  3.087 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  1.686 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  2.140 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [  3.577 s]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [  4.854 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.083 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  1.650 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  1.062 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.085 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  2.026 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.103 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  2.950 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.269 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [  7.138 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  4.366 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  1.704 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  4.262 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  2.684 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  6.478 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  1.172 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [  2.150 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  2.053 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  0.290 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  1.686 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  2.631 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  0.953 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.250 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  1.696 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  1.670 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  0.813 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  0.935 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.071 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  1.317 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  2.900 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [  1.079 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [  2.465 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  1.354 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [  3.009 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  2.571 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  4.093 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  1.597 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  1.670 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.060 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [02:29 min]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [01:11 min]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [  0.275 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [01:59 min]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [  0.190 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [  0.224 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [  0.253 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [  0.053 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  0.970 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.039 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:02 min
[INFO] Finished at: 2017-05-01T17:09:55-07:00
[INFO] Final Memory: 143M/500M
[INFO] ------------------------------------------------------------------------
{noformat}
Comparing with branch-2, build time for "Apache Hadoop Client" are increasing from 2+ s to
several minutes (for API, Runtime, etc.). Also, build of Minicluster are changing from 1.3
s to 2 minutes, etc. Still negligible change for build process? 

> Shade Guava everywhere
> ----------------------
>
>                 Key: HADOOP-14284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14284
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 3.0.0-alpha3
>            Reporter: Andrew Wang
>            Assignee: Tsuyoshi Ozawa
>            Priority: Blocker
>         Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch, HADOOP-14284.004.patch,
HADOOP-14284.007.patch, HADOOP-14284.010.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts. Unfortunately, these
projects also consume our private artifacts like {{hadoop-hdfs}}. They also are unlikely on
the new shaded client introduced by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams. This isn't
a requirement for all dependency upgrades, but it's necessary for known-bad dependencies like
Guava.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message