samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chinmay Soman <>
Subject Re: improving hello-samza / testing
Date Wed, 17 Jun 2015 01:11:55 GMT
We've built a driver program which kinda falls along approach (1) listed in
your email.

The driver program accepts a custom task object and has a way to inject
data -> which in turn invokes the process method. For now we're assuming
logical time and use the frequency of process() invocations to deduce when
to invoke the window() method (for eg: invoke window once for every 4 calls
to process).

We've also built our own Incoming and Outgoing envelope -> which is just in
the form of a Java List. This is how the result is evaluated (either get
the full result list or define a callback which is invoked every time a
collector.send is called). Its still work in progress and the goal is to
make unit testing along the lines of Storm unit tests.

On Tue, Jun 16, 2015 at 5:17 PM, Chris Riccomini <>

> Hey Tim,
> This is a really good discussion to have. The testing that I've seen with
> Samza falls into two categories:
> 1. Instantiate your StreamTask, and mock all params in the process()/init()
> methods.
> 2. A mini-ontegration test that starts ZooKeeper, and Kafka, and feeds
> messages into a topic, and validates it gets messages back out from the
> output topic.
> 3. A full blown integration test that uses Zopkio.
> For an example of (2), in practice, have a look at TestStatefuleTask:
> As you can see, writing this kind of integration test can be a bit painful.
> (3) is documented here:
> Another way to test would be to start a full-blown container using
> ThreadJobFactory/ProcessJobFactory, but use a MockSystemFactory to mock out
> the system consumer/system producer.
> Has anyone else tested Samza in other ways?
> Cheers,
> Chris
> On Tue, Jun 16, 2015 at 11:00 AM, Tim Williams <>
> wrote:
> > I'm learning samza by the hello-samza project and notice the lack of
> > tests.  Where's a good place to learn how folks are properly testing
> > things written with samza?
> >
> > Thanks,
> > --tim
> >

Thanks and regards

Chinmay Soman

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message