hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer ...@apache.org>
Subject [DISCUSS] Replacing the automatic Jenkins jobs with Apache Yetus qbt
Date Tue, 31 May 2016 18:12:49 GMT

Hey gang.

	I’ve been thinking a lot over the past few months about the various Jenkins jobs that we
have that run after a commit.  Their job is to tell us when a commit break a unit test. They
eat up an incredible amount of time on the build infrastructure and I’m not really sure
if they provide much value in their current form. I suspect most of us pretty much ignore
those emails, however, for a variety of reasons:

a) There are simply too many of them.
b) The content of the email is hard to quickly parse/read to get useful information.
c) The unit tests are almost always broken.

	Worse yet: these jobs don’t actually test everything (all of hadoop-tools is missing!)
and skip out on some vital information.

	With Apache Yetus 0.3.0, we added a mode called ‘qbt’ aka Quality Build Tool.  It effectively
run Apache Yetus over the entire source tree and generates the columnar output similar to
what you would see during precommit.  I’ve been running this mode against Apache Hadoop
on Jenkins for a few months now, working out the kinks, etc, and it was working well except
for one thing… the generated report was still not great over email:

1) Couldn’t use HTML since the ASF mailing lists reject HTML-formatted mail
2) Mail clients that used proportional fonts were still a big mess

	For 0.4.0, I’m working on a feature that adds a ‘brief’ mode  [ YETUS-390 ] with the
(long term?) intention of replacing all of these emails that get generated by Jenkins.  I’d
like to hear some feedback on if a) I’m just spinning my wheels and no one is actually interested
in fixing this problem and b) the replacement email generated either needs more info or less
info.

	Here’s the currently generated email.  Note that some subsystems have been told not to
fail the build due to warnings (e.g., cc, javac, javadoc, shellcheck, whitespace, etc) but
still generate a log of problems so that they may be investigated and used for cleanup.

	Thanks!

==== cut here =====
For more details, see https://builds.apache.org/job/hadoop-qbt/39/

[Jun 1, 2016 12:41:15 AM] (kai.zheng) Revert "HADOOP-12579. Deprecate and remove WriteableRPCEngine.




-1 overall


The following subsystems reported failures:
   findbugs unit


The following subsystems are considered long running:
   unit


   FindBugs :

      module:hadoop-common-project/hadoop-minikdc 
      org.apache.hadoop.minikdc.MiniKdc.stop() calls Thread.sleep() with a lock held At MiniKdc.java:lock
held At MiniKdc.java:[line 345] 

   Failed junit tests :

      hadoop.net.TestClusterTopology 
      hadoop.yarn.server.resourcemanager.TestAMAuthorization 
      hadoop.yarn.server.resourcemanager.TestClientRMTokens 
      hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
      hadoop.yarn.server.TestContainerManagerSecurity 
      hadoop.yarn.client.TestGetGroups 
      hadoop.yarn.client.api.impl.TestDistributedScheduling 
      hadoop.yarn.client.api.impl.TestAMRMProxy 
      hadoop.mapreduce.tools.TestCLI 
      hadoop.mapred.TestMiniMRChildTask 

   Timed out junit tests :

      org.apache.hadoop.yarn.client.cli.TestYarnCLI 
      org.apache.hadoop.yarn.client.api.impl.TestAMRMClient 
      org.apache.hadoop.yarn.client.api.impl.TestYarnClient 
      org.apache.hadoop.yarn.client.api.impl.TestNMClient 

Console output: 
   https://builds.apache.org/job/hadoop-qbt/39/console

Logs:

    https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-compile-cc-root.txt 
    https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-compile-javac-root.txt 
    https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-checkstyle-root.txt 
    https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-patch-pylint.txt 
    https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-patch-shellcheck.txt 
    https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-patch-shelldocs.txt 
    https://builds.apache.org/job/hadoop-qbt/39/artifact/out/whitespace-eol.txt 
    https://builds.apache.org/job/hadoop-qbt/39/artifact/out/whitespace-tabs.txt 
    https://builds.apache.org/job/hadoop-qbt/39/artifact/out/branch-findbugs-hadoop-common-project_hadoop-minikdc-warnings.html

    https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-javadoc-javadoc-root.txt

    https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt

    https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt

    https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt

    https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt

    https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt

    https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt

    https://builds.apache.org/job/hadoop-qbt/39/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt


Powered by Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org

==== cut here ====
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org


Mime
View raw message