apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <thomas.we...@gmail.com>
Subject How to write a unit test that runs embedded cluster and verifies results
Date Thu, 23 Jun 2016 21:18:26 GMT
Posting it here as the question comes up often. When testing multiple
operators in a DAG, how do I know that it "worked"? Following is the

  public void testSomeMethod() throws Exception
    LocalMode lma = LocalMode.newInstance();
    lma.prepareDAG(new Application(), new Configuration(false));
    LocalMode.Controller lc = lma.getController();

*    lc.run(<millis>);*

Something could have gone wrong or maybe the test never terminates and
blocks the entire test suite.

Instead, I can write it to run the DAG (asynchronously) *and* verify the
expected results by polling with a timeout. Examples:


Writes to DB, uses DB to poll for results.


Collects data in memory and terminates when expected result is found.

You can also use BaseOperator.shutdown within the operator thread and the
operator and all its downstream dependencies will terminate. When that is
done from the (only) input operator, it will terminate the entire DAG and
cause control to return to the unit test thread (when using synchronous


View raw message