Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ambari.apache.org
Content-Type: multipart/alternative;
 boundary="===============4965644874964700247=="
MIME-Version: 1.0
Subject: Re: Review Request 41569: AMBARI-14439. Categorize Ambari unit tests
 with Slow/Fast to run only desired group
From: "Alejandro Fernandez" <afernandez@hortonworks.com>
To: "Sumit Mohanty" <smohanty@hortonworks.com>,
 "Dmitro Lisnichenko" <dlysnichenko@hortonworks.com>,
 "Srimanth Gunturi" <srimanth@hortonworks.com>,
 "Robert Nettleton" <rnettleton@hortonworks.com>,
 "Aravindan Vijayan" <avijayan@hortonworks.com>,
 "Jayush Luniya" <jluniya@hortonworks.com>,
 "Nahappan Somasundaram" <nsomasundaram@hortonworks.com>,
 "Yusaku Sako" <yusaku@hortonworks.com>, "Di Li" <dili@ca.ibm.com>,
 "Sid Wagle" <swagle@hortonworks.com>,
 "Dmytro Grinenko" <hapy.lestat@gmail.com>,
 "Jaimin Jetly" <jaimin@hortonworks.com>,
 "Jonathan Hurley" <jhurley@hortonworks.com>,
 "Swapan Shridhar" <sshridhar@hortonworks.com>,
 "Nate Cole" <ncole@hortonworks.com>, "Robert Levas" <rlevas@hortonworks.com>
Cc: "Ambari" <dev@ambari.apache.org>,
 "Alejandro Fernandez" <afernandez@hortonworks.com>
Date: Mon, 28 Dec 2015 23:08:24 -0000
Message-ID: <20151228230824.4182.94874@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "Alejandro Fernandez" <noreply@reviews.apache.org>
References: <20151219221855.1646.17104@reviews.apache.org>
In-Reply-To: <20151219221855.1646.17104@reviews.apache.org>
Reply-To: "Alejandro Fernandez" <afernandez@hortonworks.com>

--===============4965644874964700247==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


> On Dec. 19, 2015, 10:18 p.m., Jonathan Hurley wrote:
> > I think we need to drop the notion of fast vs non-fast. I know that Nate has been working on re-using the same persistence unit between tests (each test gets its own cluster, own unique hosts, etc) ... and this has dramatically increased performance. 
> > 
> > With that said, I think we can still categorize and label our tests. But I think we should take a difference approach.
> > 
> > It's cool to group them logically like Alerts, Kerberos, ORM, Upgrade, etc. 
> > 
> > But we should also create groupings based on what the tests are doing. Something like:
> > SmallTest - accesses only mocks, most used for direct unit tests of single functional aspects
> > MediumTest - works with mocks and DB, sets up cluster and some data for a single unit of test
> > LargeTest - an end-to-end style test (like testing an upgrade or downgrade)
> > 
> > Even more, we can group them for quick runs from the command by mixing and matching above logical groupings. Consider:
> > MAT (Minimum Acceptance Tests): Basic functionality that must pass (like cluster creation, adding services, pure ORM)
> > SRG (Short Regression): A mixture of tests from above logical units (Alerts, Kerberos, Upgrade) which don't take too long
> > LRG (Long Regression): A mixture of tests from above logical units (Alerts, Kerberos, Upgrade) which are known to take a long time
> 
> Nate Cole wrote:
>     +1 to this.
> 
> Dmitro Lisnichenko wrote:
>     Are not SmallTest/MediumTest/LargeTest something like unit/integration/functional tests? I'd say that a big part of our tests consists of over-detailed functional tests, that tend to test too much and that's why break on any implementation change like mock method invocation count. So I'd strongly +1 to clear separation of roles you are proposing. Behaviour
>     
>     Also, it would be great to have some commonly-used mock implementations like cluster with few hosts and services as a separate class, so we could skip defining something like createCluster() at every test suite.
> 
> Dmitro Lisnichenko wrote:
>     * I mean that functional/integration tests should test only high-level behaviour, and mock out any logic that is covered by unit tests. So tests are more loosely coupled
> 
> Jayush Luniya wrote:
>     +1 on the suggestions above.

I like these suggestions. Since performance can always change, I'm ok with renaming them as
Unit Test - Very simple mocks, in the order of 100 ms or faster
Integration Test - multiple classes being tested together, in the order of 1000 ms or faster
Functional Test - heavy lifting that tests features (anything that has to setup a DB, add cluster, services, hosts, perform RUs, etc). Some of these can take less than 1000 ms, but for sure if it takes longer than 1000 ms it is likely in this category.

In additional, I will tag them with the feature, e.g., Stack Upgrade, Security (Kerberos), Blueprints, etc.
This way, we can use the test runner to be very specific, e.g., Stack Upgrade && Unit Test.


- Alejandro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41569/#review111389
-----------------------------------------------------------


On Dec. 18, 2015, 10:26 p.m., Alejandro Fernandez wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41569/
> -----------------------------------------------------------
> 
> (Updated Dec. 18, 2015, 10:26 p.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jaimin Jetly, Jonathan Hurley, Jayush Luniya, Nate Cole, Robert Levas, Robert Nettleton, Nahappan Somasundaram, Sumit Mohanty, Srimanth Gunturi, Swapan Shridhar, Sid Wagle, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-14439
>     https://issues.apache.org/jira/browse/AMBARI-14439
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Categorize the unit tests with wither SlowTest, FastTest, and some other group based on the area.
> This will allow us to run a test suite that matches a given group.
> 
>     mvn clean test -P ${profile_id}
>     mvn clean -Drat.ignoreErrors=true test -P FastTest > fast-tests.txt
>     mvn clean -Drat.ignoreErrors=true test -P NonFastTest > nonfast-tests.txt
> 
> 
> Diffs
> -----
> 
>   ambari-server/pom.xml 56d9e44 
>   ambari-server/src/test/java/org/apache/ambari/server/AlertTest.java PRE-CREATION 
>   ambari-server/src/test/java/org/apache/ambari/server/AmbariUpgradeTest.java PRE-CREATION 
>   ambari-server/src/test/java/org/apache/ambari/server/BlueprintTest.java PRE-CREATION 
>   ambari-server/src/test/java/org/apache/ambari/server/FastTest.java PRE-CREATION 
>   ambari-server/src/test/java/org/apache/ambari/server/KerberosTest.java PRE-CREATION 
>   ambari-server/src/test/java/org/apache/ambari/server/SlowTest.java PRE-CREATION 
>   ambari-server/src/test/java/org/apache/ambari/server/StackUpgradeTest.java PRE-CREATION 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/ClientRetryPropertyCheckTest.java 7b8239c 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/ComponentsInstallationCheckTest.java 450d74e 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/ConfigurationMergeCheckTest.java 68a0522 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/HiveDynamicServiceDiscoveryCheckTest.java cdf13eb 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/HiveMultipleMetastoreCheckTest.java 16f383a 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/HostMaintenanceModeCheckTest.java 0e14376 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/HostsHeartbeatCheckTest.java cc2c276 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/HostsMasterMaintenanceCheckTest.java 9fcb319 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/HostsRepositoryVersionCheckTest.java 4529554 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/InstallPackagesCheckTest.java 080ca3a 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/MapReduce2JobHistoryStatePreservingCheckTest.java bfe0c3e 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/RangerPasswordCheckTest.java afa3789 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/SecondaryNamenodeDeletedCheckTest.java e2617bf 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/ServicesMaintenanceModeCheckTest.java a941b7a 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/ServicesMapReduceDistributedCacheCheckTest.java 22f2b1b 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/ServicesNamenodeHighAvailabilityCheckTest.java abe7abe 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/ServicesNamenodeTruncateCheckTest.java 87d4167 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/ServicesUpCheckTest.java 88826a0 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/ServicesYarnWorkPreservingCheckTest.java 98cfb18 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/UpgradeCheckOrderTest.java 7d70311 
>   ambari-server/src/test/java/org/apache/ambari/server/checks/YarnTimelineServerStatePreservingCheckTest.java 7469bbc 
> 
> Diff: https://reviews.apache.org/r/41569/diff/
> 
> 
> Testing
> -------
> 
> Verified it worked on the handful of tests I annotated.
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>


--===============4965644874964700247==--