Mailing-List: contact dev-help@samza.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@samza.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CABYbY7ezog9NVhKbPdsD8UWKJoCOp_+DVq65+Oa534Hp-jf2Vw@mail.gmail.com>
References: 
 <CAG_bHowhrHPfaeHz0TK_gaep7faPX0mu5XsGVF=oaUrpKafuCw@mail.gmail.com>
	<CABYbY7ezog9NVhKbPdsD8UWKJoCOp_+DVq65+Oa534Hp-jf2Vw@mail.gmail.com>
Date: Tue, 16 Jun 2015 18:11:55 -0700
Message-ID: 
 <CAMrSGZWjNJRTAk5CBOixc2h1g0jVf7LhNWoUAgQQ1Zt+nCnnzQ@mail.gmail.com>
Subject: Re: improving hello-samza / testing
From: Chinmay Soman <chinmay.cerebro@gmail.com>
To: dev@samza.apache.org
Content-Type: multipart/alternative; boundary=001a1141bd967ab5880518ac6068

--001a1141bd967ab5880518ac6068
Content-Type: text/plain; charset=UTF-8

We've built a driver program which kinda falls along approach (1) listed in
your email.

The driver program accepts a custom task object and has a way to inject
data -> which in turn invokes the process method. For now we're assuming
logical time and use the frequency of process() invocations to deduce when
to invoke the window() method (for eg: invoke window once for every 4 calls
to process).

We've also built our own Incoming and Outgoing envelope -> which is just in
the form of a Java List. This is how the result is evaluated (either get
the full result list or define a callback which is invoked every time a
collector.send is called). Its still work in progress and the goal is to
make unit testing along the lines of Storm unit tests.

On Tue, Jun 16, 2015 at 5:17 PM, Chris Riccomini <criccomini@apache.org>
wrote:

> Hey Tim,
>
> This is a really good discussion to have. The testing that I've seen with
> Samza falls into two categories:
>
> 1. Instantiate your StreamTask, and mock all params in the process()/init()
> methods.
> 2. A mini-ontegration test that starts ZooKeeper, and Kafka, and feeds
> messages into a topic, and validates it gets messages back out from the
> output topic.
> 3. A full blown integration test that uses Zopkio.
>
> For an example of (2), in practice, have a look at TestStatefuleTask:
>
>
>
> https://git-wip-us.apache.org/repos/asf?p=samza.git;a=blob;f=samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala;h=ea702a919348305ff95ce0b4ca1996a13aff04ec;hb=HEAD
>
> As you can see, writing this kind of integration test can be a bit painful.
>
> (3) is documented here:
>
>   http://samza.apache.org/contribute/tests.html
>
> Another way to test would be to start a full-blown container using
> ThreadJobFactory/ProcessJobFactory, but use a MockSystemFactory to mock out
> the system consumer/system producer.
>
> Has anyone else tested Samza in other ways?
>
> Cheers,
> Chris
>
> On Tue, Jun 16, 2015 at 11:00 AM, Tim Williams <williamstw@gmail.com>
> wrote:
>
> > I'm learning samza by the hello-samza project and notice the lack of
> > tests.  Where's a good place to learn how folks are properly testing
> > things written with samza?
> >
> > Thanks,
> > --tim
> >
>


-- 
Thanks and regards

Chinmay Soman

--001a1141bd967ab5880518ac6068--