apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Isha Arkatkar <i...@datatorrent.com>
Subject Re: StreamingAppMasterService Unit testing
Date Tue, 06 Oct 2015 22:10:28 GMT
Hi ,

   I went ahead with Mockito for testing StreamingAppMasterService to
check blacklisting of consecutively failing nodes. Here is a pull request
for the same:
https://github.com/apache/incubator-apex-core/pull/98

  Mockito was fairly easy to use for  for mocking responses from Hadoop
APIs and some of StreamingAppMasterService methods. I had captured some
basic Mocktio APIs that I used in this document (in case someone has a use
for it):
https://docs.google.com/document/d/1WBdWceHNIVg4UeTL06pqzoJLQQ-Q7yLNzTV1IDoHFyo/edit?usp=sharing

Thanks,
Isha


On Wed, Sep 30, 2015 at 3:49 PM, Isha Arkatkar <isha@datatorrent.com> wrote:

> I was thinking of mocking the calls that we make during allocation. E.g.
> RegisterApplicationMasterResponse, AMRMClient.allocate call,
> AllocateResponse returned etc..
> But yes, if there is any supporting library for the same, it would be very
> useful!
>
> Isha
>
> On Wed, Sep 30, 2015 at 1:26 PM, Pramod Immaneni <pramod@datatorrent.com>
> wrote:
>
>> Would going to Mockito route mean we have to keep up with the RMClient
>> protocol and handle generation of responses ourselves, are there any
>> supporting libraries in hadoop that would help us with that.
>>
>> On Wed, Sep 30, 2015 at 12:36 PM, Isha Arkatkar <isha@datatorrent.com>
>> wrote:
>>
>> > Hi all,
>> >
>> >        I want to add some unit testing for container allocation flow in
>> > StreamingAppMasterService class. Some of the scenarios like consecutive
>> > node failures etc. are hard to simulate manually and would be nice to
>> cover
>> > that through unit testing.
>> >
>> >     I am right now exploring 2 options: Mocking RMClient responses using
>> > Mockito and via MiniYARNCluster.
>> >
>> >    Mockito route requires some refactoring in
>> StreamingAppMasterService. If
>> > we divide execute method in multiple functions, then those can be tested
>> > individually. It gives more flexibility to simulate failure scenarios.
>> >
>> >   With MiniYARNCluster, we have a few tests in StreamMiniClusterTest.
>> These
>> > are probably good end-to-end tests. But I don't know if we can set
>> > container launch failures.
>> >
>> >   Please suggest which is best way to go :)
>> >
>> > Thanks!
>> > Isha
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message