Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 11 Sep 2017 16:25:00 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13081084.1497955168000.80421.1505147100667@Atlassian.JIRA>
In-Reply-To: <JIRA.13081084.1497955168000@Atlassian.JIRA>
References: <JIRA.13081084.1497955168000@Atlassian.JIRA> <JIRA.13081084.1497955168269@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HADOOP-14553) Add (parallelized) integration
 tests to hadoop-azure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Mon, 11 Sep 2017 16:25:08 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-14553?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16=
161518#comment-16161518 ]=20

Hadoop QA commented on HADOOP-14553:
------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 11s=
{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0=
m  0s{color} | {color:green} The patch does not contain any @author tags. {=
color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}=
  0m  0s{color} | {color:green} The patch appears to include 103 new or mod=
ified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 17s=
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}=
 14m 52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18=
m 30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}=
  2m 30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1=
m 58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  =
2m 37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1=
m 29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 19s=
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}=
  1m 19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14=
m  6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m =
 6s{color} | {color:green} root generated 0 new + 1281 unchanged - 2 fixed =
=3D 1281 total (was 1283) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:oran=
ge}  2m 19s{color} | {color:orange} root: The patch generated 123 new + 211=
 unchanged - 177 fixed =3D 334 total (was 388) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1=
m 44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  =
0s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Us=
e git apply --whitespace=3Dfix <<patch_file>>. Refer https://git-scm.com/do=
cs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  2=
s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  =
3m  8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1=
m 20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m  0s{col=
or} | {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 1=
0s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}=
  0m 34s{color} | {color:green} The patch does not generate ASF License war=
nings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 56s{colo=
r} | {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14553 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/1288643=
3/HADOOP-14553-016.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsit=
e  unit  findbugs  checkstyle  xml  |
| uname | Linux 8e7587899bd2 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 =
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provide=
d.sh |
| git revision | trunk / 5ab21df |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13254/a=
rtifact/patchprocess/diff-checkstyle-root.txt |
| whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/13254/a=
rtifact/patchprocess/whitespace-eol.txt |
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/13254/artifac=
t/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt |
|  Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/1325=
4/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azur=
e U: . |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/132=
54/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.


> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>
>                 Key: HADOOP-14553
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14553
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, HADO=
OP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, HADOOP-=
14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, HADOOP-145=
53-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch, HADOOP-14553-=
012.patch, HADOOP-14553-014.patch, HADOOP-14553-015.patch, HADOOP-14553-016=
.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all c=
alled Test* there's no clear differentiation from unit tests which Jenkins =
can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, paralleli=
ze (which includes having separate paths for every test suite). The code in=
 hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*:  Adding a list of requirements to clarify the ac=
ceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins,=
 ii) IntegrationTests - requires Azure Storage account, executed by enginee=
rs prior to check-in, and if needed, iii)=09ScaleTests =E2=80=93 long runni=
ng performance and scalability tests.
> # Define configuration profiles to run tests with different settings.  Al=
lows an engineer to run =E2=80=9CIntegrationTests=E2=80=9D with fs.azure.se=
cure.mode =3D true and false.  Need to review settings to see what else wou=
ld benefit.
> # Maven commands to run b) and c).  Turns out it is not easy to do with M=
aven, so we might have to run it multiple times to run with different confi=
guration settings.
> # Document how to add and run tests and the process for contributing to A=
pache Hadoop.  Steve shared an example at https://github.com/apache/hadoop/=
blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testi=
ng.md=20
> # UnitTests should run in under 2 minutes and IntegrationTests should run=
 in under 15 minutes, even on slower network connections.  (These are rough=
 goals)
> # Ensure test data (containers/blobs/etc) is deleted.  Exceptions for lar=
ge persistent content used repeatedly to expedite test execution.=20


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org