reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergiy Matusevych <sergiy.matusev...@gmail.com>
Subject Re: Questions on running REEF-TESTS
Date Fri, 23 Feb 2018 00:00:12 GMT
Hi Chengxi,

Yes, the idea is that each unit test in Java must belong to some test
suite, which, in turn, can be a part of some other suite, and, eventually,
the AllTestsSuite. That is, our unit tests should form a hierarchy, and
each test *must* be traceable from AllTestsSuite. Unfortunately, I do not
know if there exists an automatic way to enforce such hierarchy, so we have
to rely on the developer's discipline and hope that whenever someone writes
a new test, this test will be included into a test suite.

I think it is a good time to review our unit tests and make sure that each
test is included in some test suite and can be run from AllTestsSuite. This
is especially important for the tests that are meant to run outside of the
IDE environment - e.g. on YARN or Azure Batch.

Cheers,
Sergiy.



On Thu, Feb 22, 2018 at 3:21 PM, Chenxi Zhao <
Chenxi.Zhao@microsoft.com.invalid> wrote:

> Thanks Sergiy!
>
> I still have one related question on the TEST suites. Could you or someone
> help clarify a bit?
>
> I am able to run the test https://github.com/apache/
> reef/blob/master/lang/java/reef-tests/src/test/java/org/apache/reef/tests/
> AllTestsSuite.java with the script you mentioned to add shaded jar to
> verify Azure Batch runtime implementation and am getting positive result.
>
> However, in order to make sure I am working towards a good test coverage -
> what is exact "End to End" test do we use to verify a runtime? I assume
> AllTestsSuite.java is it but want to check in case there is a gap.
>
> Thanks,
> Chenxi
>
>
>
> -----Original Message-----
> From: Sergiy Matusevych <sergiy.matusevych@gmail.com>
> Sent: Friday, February 16, 2018 6:19 PM
> To: dev@reef.apache.org
> Subject: Re: Questions on running REEF-TESTS
>
> Hi Chenxi,
>
> You can take a look at the script reef/bin/runtests.ps1
>
> If I remember correctly, you have to specify the shaded jar with your app
> (e.g. examples) as well as the test jar - something along the lines of
>
> .\bin\runtests.ps1 -VerboseLog -Jars
> ".\lang\java\reef-examples\target\reef-examples-0.17.0-
> SNAPSHOT-shaded.jar;.\lang\java\reef-tests\target\reef-
> tests-0.17.0-SNAPSHOT-test-jar-with-dependencies.jar"
> org.apache.reef.tests.examples.TestHelloREEF
>
> You can also specify the -Yarn option to run it on YARN, if your
> hadoop/etc/ and $HADOOP_HOME are set correctly.
>
> Cheers,
> Sergiy.
>
>
>
> On Thu, Feb 15, 2018 at 4:27 PM, Chenxi Zhao < Chenxi.Zhao@microsoft.com.invalid>
> wrote:
>
> > Hi -
> >
> > I am trying to verify AzureBatch Runtime implementation through
> > REEF-TESTS ("reef/lang/java/reef-tests/mvn test"), but I got several
> > problems on REEF-TESTS itself. Could someone help clarify them:
> >
> >
> >   1.  reef-examples-{$REEF_VERSION}-SNAPSHOT-shaded.jar is not
> > included as library in reef/global folder. Instead, it contains a test
> > Jar which has the test classes instead.
> >
> > How does this test find REEFLauncher.class or DriverLauncher.class, as
> > they are not pointed in the ClassPath? Especially like testing a
> > remote environment, like YARN, Azure Batch.
> >
> >
> >   1.  I tried to run REEF-TEST in YARN environment by giving the
> > environment variable in
> > (https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> > ub.com%2Fapache%2F&data=04%7C01%7CChenxi.Zhao%40microsoft.com%7C73af41
> > ba5101428bc69d08d575acd79a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%
> > 7C636544307568080515%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQI
> > joiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-1&sdata=N3YXyonTKdSKt9CIOX6wNFj
> > 5PMHKDKAoRxF6W9W%2BCOU%3D&reserved=0
> > reef/blob/master/lang/java/reef-tests/src/test/java/org/apache/reef/te
> > sts/
> > TestEnvironmentFactory.java#L38)
> >
> > However it give exceptions and the test could not pass:
> >
> >
> >
> > java.lang.RuntimeException: org.apache.reef.tang.
> exceptions.InjectionException:
> > Could not invoke constructor: new JobUploader(YarnConfiguration = new
> > YarnConfigurationConstructor(String FileSystemUrl = NULL),
> > JobSubmissionDirectoryProvider = new
> > JobSubmissionDirectoryProviderImpl(String
> > JobSubmissionDirectoryPrefix = /tmp, new REEFFileNames()))
> >
> >
> >
> > And I didn't find any place in YARN test that asks cluster's
> > credential and URL, or taking parameters from files as well. I wonder
> > how this is designed to run. Do we have any document tells the
> configuration?
> >
> >
> > Thanks,
> > Chenxi
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message