Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 4 Aug 2017 17:39:01 +0000 (UTC)
From: "Thomas Marquardt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13081084.1497955168000.95742.1501868341305@Atlassian.JIRA>
In-Reply-To: <JIRA.13081084.1497955168000@Atlassian.JIRA>
References: <JIRA.13081084.1497955168000@Atlassian.JIRA> <JIRA.13081084.1497955168269@jira-lw-us.apache.org>
Subject: [jira] [Updated] (HADOOP-14553) Add (parallelized) integration
 tests to hadoop-azure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Fri, 04 Aug 2017 17:39:10 -0000


     [ https://issues.apache.org/jira/browse/HADOOP-14553?page=3Dcom.atlass=
ian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Marquardt updated HADOOP-14553:
--------------------------------------
    Description:=20
The Azure tests are slow to run as they are serialized, as they are all cal=
led Test* there's no clear differentiation from unit tests which Jenkins ca=
n run, and integration tests which it can't.

Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize=
 (which includes having separate paths for every test suite). The code in h=
adoop-aws's POM &c show what to do.

*UPDATE August 4, 2017*:  Adding a list of requirements to clarify the acce=
ptance criteria for this JIRA:
# Parallelize test execution
# Define test groups: i) UnitTests - self-contained, executed by Jenkins, i=
i) IntegrationTests - requires Azure Storage account, executed by engineers=
 prior to check-in, and if needed, iii)=09ScaleTests =E2=80=93 long running=
 performance and scalability tests.
# Define configuration profiles to run tests with different settings.  Allo=
ws an engineer to run =E2=80=9CIntegrationTests=E2=80=9D with fs.azure.secu=
re.mode =3D true and false.  Need to review settings to see what else would=
 benefit.
# Maven commands to run b) and c).  Turns out it is not easy to do with Mav=
en, so we might have to run it multiple times to run with different configu=
ration settings.
# Document how to add and run tests and the process for contributing to Apa=
che Hadoop.  Steve shared an example at https://github.com/apache/hadoop/bl=
ob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing=
.md=20
# UnitTests should run in under 2 minutes and IntegrationTests should run i=
n under 15 minutes, even on slower network connections.  (These are rough g=
oals)
# Ensure test data (containers/blobs/etc) is deleted.  Exceptions for large=
 persistent content used repeatedly to expedite test execution.=20


  was:
The Azure tests are slow to run as they are serialized, as they are all cal=
led Test* there's no clear differentiation from unit tests which Jenkins ca=
n run, and integration tests which it can't.

Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize=
 (which includes having separate paths for every test suite). The code in h=
adoop-aws's POM &c show what to do.


> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>
>                 Key: HADOOP-14553
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14553
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, HADO=
OP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, HADOOP-=
14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, HADOOP-145=
53-009.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all c=
alled Test* there's no clear differentiation from unit tests which Jenkins =
can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, paralleli=
ze (which includes having separate paths for every test suite). The code in=
 hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*:  Adding a list of requirements to clarify the ac=
ceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins,=
 ii) IntegrationTests - requires Azure Storage account, executed by enginee=
rs prior to check-in, and if needed, iii)=09ScaleTests =E2=80=93 long runni=
ng performance and scalability tests.
> # Define configuration profiles to run tests with different settings.  Al=
lows an engineer to run =E2=80=9CIntegrationTests=E2=80=9D with fs.azure.se=
cure.mode =3D true and false.  Need to review settings to see what else wou=
ld benefit.
> # Maven commands to run b) and c).  Turns out it is not easy to do with M=
aven, so we might have to run it multiple times to run with different confi=
guration settings.
> # Document how to add and run tests and the process for contributing to A=
pache Hadoop.  Steve shared an example at https://github.com/apache/hadoop/=
blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testi=
ng.md=20
> # UnitTests should run in under 2 minutes and IntegrationTests should run=
 in under 15 minutes, even on slower network connections.  (These are rough=
 goals)
> # Ensure test data (containers/blobs/etc) is deleted.  Exceptions for lar=
ge persistent content used repeatedly to expedite test execution.=20


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org