Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 10 Aug 2017 23:06:00 +0000 (UTC)
From: "Esfandiar Manii (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13081084.1497955168000.5334.1502406360525@Atlassian.JIRA>
In-Reply-To: <JIRA.13081084.1497955168000@Atlassian.JIRA>
References: <JIRA.13081084.1497955168000@Atlassian.JIRA> <JIRA.13081084.1497955168269@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (HADOOP-14553) Add (parallelized)
 integration tests to hadoop-azure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 10 Aug 2017 23:06:06 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-14553?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16=
122497#comment-16122497 ]=20

Esfandiar Manii edited comment on HADOOP-14553 at 8/10/17 11:05 PM:
--------------------------------------------------------------------

Actually=20
{code:java}
"${test.build.data.replace('\\', '\\\\')}",
  "${test.build.dir.replace('\\', '\\\\')}",
  "${hadoop.tmp.dir.replace('\\', '\\\\')}"];=20
{code}

is not working well. It seems it created wrong directories under hadoop-azu=
re. Is there any easy way to print two back slashes through pom.xml configu=
rations?

Currently the issue is you are doing

{code:java}
var baseDirs =3D [
  "${test.build.data}",
  "${test.build.dir}",
  "${hadoop.tmp.dir}"]
{code}


in windows if test.build.data is f:\a\b
in JS you will have
baseDirs =3D [ 'f'] since \a \b are escape sequence chars and the path is i=
nvalid in Windows.=20


was (Author: esmanii):
Actually "${test.build.data.replace('\\', '\\\\')}",
  "${test.build.dir.replace('\\', '\\\\')}",
  "${hadoop.tmp.dir.replace('\\', '\\\\')}"];=20
is not working well. It seems it created wrong directories under hadoop-azu=
re. Is there any easy way to print two back slashes through pom.xml configu=
rations?

Currently the issue is you are doing
var baseDirs =3D [
  "${test.build.data}",
  "${test.build.dir}",
  "${hadoop.tmp.dir}"]

in windows if test.build.data is f:\a\b
in JS you will have
baseDirs =3D [ 'f'] since \a \b are escape sequence chars and the path is i=
nvalid in Windows.=20


> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>
>                 Key: HADOOP-14553
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14553
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, HADO=
OP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, HADOOP-=
14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, HADOOP-145=
53-009.patch, HADOOP-14553-010.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all c=
alled Test* there's no clear differentiation from unit tests which Jenkins =
can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, paralleli=
ze (which includes having separate paths for every test suite). The code in=
 hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*:  Adding a list of requirements to clarify the ac=
ceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins,=
 ii) IntegrationTests - requires Azure Storage account, executed by enginee=
rs prior to check-in, and if needed, iii)=09ScaleTests =E2=80=93 long runni=
ng performance and scalability tests.
> # Define configuration profiles to run tests with different settings.  Al=
lows an engineer to run =E2=80=9CIntegrationTests=E2=80=9D with fs.azure.se=
cure.mode =3D true and false.  Need to review settings to see what else wou=
ld benefit.
> # Maven commands to run b) and c).  Turns out it is not easy to do with M=
aven, so we might have to run it multiple times to run with different confi=
guration settings.
> # Document how to add and run tests and the process for contributing to A=
pache Hadoop.  Steve shared an example at https://github.com/apache/hadoop/=
blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testi=
ng.md=20
> # UnitTests should run in under 2 minutes and IntegrationTests should run=
 in under 15 minutes, even on slower network connections.  (These are rough=
 goals)
> # Ensure test data (containers/blobs/etc) is deleted.  Exceptions for lar=
ge persistent content used repeatedly to expedite test execution.=20


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org