samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chinmay Soman <chinmay.cere...@gmail.com>
Subject Re: improving hello-samza / testing
Date Wed, 17 Jun 2015 01:11:55 GMT
We've built a driver program which kinda falls along approach (1) listed in
your email.

The driver program accepts a custom task object and has a way to inject
data -> which in turn invokes the process method. For now we're assuming
logical time and use the frequency of process() invocations to deduce when
to invoke the window() method (for eg: invoke window once for every 4 calls
to process).

We've also built our own Incoming and Outgoing envelope -> which is just in
the form of a Java List. This is how the result is evaluated (either get
the full result list or define a callback which is invoked every time a
collector.send is called). Its still work in progress and the goal is to
make unit testing along the lines of Storm unit tests.

On Tue, Jun 16, 2015 at 5:17 PM, Chris Riccomini <criccomini@apache.org>
wrote:

> Hey Tim,
>
> This is a really good discussion to have. The testing that I've seen with
> Samza falls into two categories:
>
> 1. Instantiate your StreamTask, and mock all params in the process()/init()
> methods.
> 2. A mini-ontegration test that starts ZooKeeper, and Kafka, and feeds
> messages into a topic, and validates it gets messages back out from the
> output topic.
> 3. A full blown integration test that uses Zopkio.
>
> For an example of (2), in practice, have a look at TestStatefuleTask:
>
>
>
> https://git-wip-us.apache.org/repos/asf?p=samza.git;a=blob;f=samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala;h=ea702a919348305ff95ce0b4ca1996a13aff04ec;hb=HEAD
>
> As you can see, writing this kind of integration test can be a bit painful.
>
> (3) is documented here:
>
>   http://samza.apache.org/contribute/tests.html
>
> Another way to test would be to start a full-blown container using
> ThreadJobFactory/ProcessJobFactory, but use a MockSystemFactory to mock out
> the system consumer/system producer.
>
> Has anyone else tested Samza in other ways?
>
> Cheers,
> Chris
>
> On Tue, Jun 16, 2015 at 11:00 AM, Tim Williams <williamstw@gmail.com>
> wrote:
>
> > I'm learning samza by the hello-samza project and notice the lack of
> > tests.  Where's a good place to learn how folks are properly testing
> > things written with samza?
> >
> > Thanks,
> > --tim
> >
>



-- 
Thanks and regards

Chinmay Soman

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message