Mailing-List: contact users-help@apex.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@apex.apache.org
MIME-Version: 1.0
From: Thomas Weise <thomas.weise@gmail.com>
Date: Thu, 23 Jun 2016 14:18:26 -0700
Message-ID: <CA+5xAo3XPThwG1oVt42QXZ1=aZMNG8OAZpVFuG_y=t0f6QHxeg@mail.gmail.com>
Subject: How to write a unit test that runs embedded cluster and verifies results
To: users@apex.apache.org
Content-Type: multipart/alternative; boundary=001a11425e3e5436050535f898a6
archived-at: Thu, 23 Jun 2016 21:18:36 -0000

--001a11425e3e5436050535f898a6
Content-Type: text/plain; charset=UTF-8

Posting it here as the question comes up often. When testing multiple
operators in a DAG, how do I know that it "worked"? Following is the
anti-pattern:

  @Test
  public void testSomeMethod() throws Exception
  {
    LocalMode lma = LocalMode.newInstance();
    lma.prepareDAG(new Application(), new Configuration(false));
    LocalMode.Controller lc = lma.getController();

*    lc.run(<millis>);*
  }

Something could have gone wrong or maybe the test never terminates and
blocks the entire test suite.

Instead, I can write it to run the DAG (asynchronously) *and* verify the
expected results by polling with a timeout. Examples:

https://github.com/DataTorrent/examples/blob/master/tutorials/exactly-once/src/test/java/com/example/myapexapp/ApplicationTest.java

Writes to DB, uses DB to poll for results.

https://github.com/DataTorrent/examples/blob/master/tutorials/unifiers/src/test/java/com/example/myapexapp/ApplicationTest.java

Collects data in memory and terminates when expected result is found.

You can also use BaseOperator.shutdown within the operator thread and the
operator and all its downstream dependencies will terminate. When that is
done from the (only) input operator, it will terminate the entire DAG and
cause control to return to the unit test thread (when using synchronous
run).

Thomas

--001a11425e3e5436050535f898a6
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Posting it here as the question comes up often. When =
testing multiple operators in a DAG, how do I know that it &quot;worked&quo=
t;? Following is the anti-pattern:</div><div><br></div><div><div>=C2=A0 @Te=
st</div><div>=C2=A0 public void testSomeMethod() throws Exception</div><div=
>=C2=A0 {</div><div>=C2=A0 =C2=A0 LocalMode lma =3D LocalMode.newInstance()=
;</div><div>=C2=A0 =C2=A0 lma.prepareDAG(new Application(),=C2=A0new Config=
uration(false));<br></div><div>=C2=A0 =C2=A0 LocalMode.Controller lc =3D lm=
a.getController();</div><div><b>=C2=A0 =C2=A0 lc.run(&lt;millis&gt;);<br></=
b></div><div>=C2=A0 }<br></div></div><div><br></div><div>Something could ha=
ve gone wrong or maybe the test never terminates and blocks the entire test=
 suite.=C2=A0<br></div><div><br></div><div>Instead, I can write it to run t=
he DAG (asynchronously)=C2=A0<b>and</b> verify the expected results by poll=
ing with a timeout. Examples:</div><div><br></div><div><a href=3D"https://g=
ithub.com/DataTorrent/examples/blob/master/tutorials/exactly-once/src/test/=
java/com/example/myapexapp/ApplicationTest.java">https://github.com/DataTor=
rent/examples/blob/master/tutorials/exactly-once/src/test/java/com/example/=
myapexapp/ApplicationTest.java</a><br></div><div><br></div><div>Writes to D=
B, uses DB to poll for results.</div><div><br></div><div><a href=3D"https:/=
/github.com/DataTorrent/examples/blob/master/tutorials/unifiers/src/test/ja=
va/com/example/myapexapp/ApplicationTest.java">https://github.com/DataTorre=
nt/examples/blob/master/tutorials/unifiers/src/test/java/com/example/myapex=
app/ApplicationTest.java</a><br></div><div><br></div><div>Collects data in =
memory and terminates when expected result is found.</div><div><br></div><d=
iv>You can also use BaseOperator.shutdown within the operator thread and th=
e operator and all its downstream dependencies will terminate. When that is=
 done from the (only) input operator, it will terminate the entire DAG and =
cause control to return to the unit test thread (when using synchronous run=
).</div><div><br></div><div>Thomas</div><div><br></div><div><br></div></div=
>

--001a11425e3e5436050535f898a6--