flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@apache.org>
Subject Re: Tests - Unit Tests versus Integration Tests
Date Mon, 21 Sep 2015 05:06:18 GMT
I agree with Ufuk. Reworking tests might cause a lot of problems! We
should do this with great care only!

The most important thing is to pay attention to open PRs and ask to
write unit tests. I would also suggest, to have at least one new unit
test per fixed bug (that of course test exactly the bug that was fixed ;))

Furthermore, I expect some people to have problems to mock correctly. Is
a little tricky sometimes. Do we have a few nice mock examples that we
could add to the contributor guide?

-Matthias


On 09/19/2015 06:39 PM, Chiwan Park wrote:
> I just created a JIRA issue [1].
> 
> Regards,
> Chiwan Park
> 
> [1] https://issues.apache.org/jira/browse/FLINK-2712
> 
>> On Sep 20, 2015, at 1:33 AM, Chiwan Park <chiwanpark@apache.org> wrote:
>>
>> Okay, I’ll create a JIRA issue and send a pull request for it. :)
>>
>> Regards,
>> Chiwan Park
>>
>>> On Sep 19, 2015, at 7:35 PM, Ufuk Celebi <uce@apache.org> wrote:
>>>
>>> Thanks Stephan for pointing this out. I agree with you. +1
>>>
>>> @Chiwan: Good idea with the Wiki. Actually maybe even better to add it to the
contribution guide? Do you have time to open a PR with Stephan’s suggestions?
>>>
>>> @Martin: I agree that it does not suffice to just point this out as a new guideline.
I think the main problem is that it is very time consuming and error prone. We have seen some
“minor refactorings” lately, which looked harmless, but actually introduced bugs. This
is a danger as well with refactoring tests (we refactor them, but don’t have the same amount
of test coverage, which results in bugs at some point in time in the future).
>>>
>>> Are there any known “heavy hitters”, which take a lot of time, but which
could be tested in unit tests instead? I would start with those if you want to do it. But
in general I would do this incrementally instead of aiming for a complete rewrite.
>>>
>>> – Ufuk
>>>
>>>> On 19 Sep 2015, at 10:53, Martin Liesenberg <martin.liesenberg@gmail.com>
wrote:
>>>>
>>>> Should there be a concerted effort to reduce the amount of unnecessary
>>>> integration tests and cover those cases by unit tests?
>>>>
>>>> We could collect the cases in a ticket and work through the list one by
>>>> one, no?
>>>>
>>>> Best regards,
>>>> Martin
>>>>
>>>> Chiwan Park <chiwanpark@apache.org> schrieb am Fr., 18. Sep. 2015 um
>>>> 12:33 Uhr:
>>>>
>>>>> Hi Stephan,
>>>>>
>>>>> Thanks for nice guide! I think we can upload this to the wiki or how
to
>>>>> contribute documentation.
>>>>> This guide would be helpful for newcomers.
>>>>>
>>>>> Regards,
>>>>> Chiwan Park
>>>>>
>>>>>> On Sep 17, 2015, at 9:33 PM, Stephan Ewen <sewen@apache.org>
wrote:
>>>>>>
>>>>>> Hi all!
>>>>>>
>>>>>> The build time of Flink with all tests is nearing 1h on Travis for
the
>>>>>> shortest run.
>>>>>> It is good that we do excessive testing, there are many mechanisms
that
>>>>>> need that.
>>>>>>
>>>>>> I have also seen that a lot of fixes that could be tested in a UnitTest
>>>>>> style are actually tested as a full Flink program (Integration test
>>>>> style)
>>>>>>
>>>>>> While these tests are always easier to write, they have two problems:
>>>>>> - The bring up the build time by about 5 secs per test
>>>>>> - They are often not as targeted to the problem as a UnitTest
>>>>>>
>>>>>> I would like to encourage everyone to keep this in mind and do Unit
tests
>>>>>> in the cases where they are the preferred choice. Please also keep
that
>>>>> in
>>>>>> mind when reviewing pull requests.
>>>>>>
>>>>>> For Example:
>>>>>> - API / TypeInformation changes can be very well tested without running
>>>>>> the program. Simply create the program and test the operator's type
info.
>>>>>> - Custom functions can be very well tested in isolation
>>>>>> - Input/Output formats actually test well in UnitTests.
>>>>>>
>>>>>> Integration tests need to be used when verifying behavior across
>>>>> components
>>>>>> / layers, so keep using them when they need to be used.
>>>>>>
>>>>>>
>>>>>> Greetings,
>>>>>> Stephan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>
>>
>>
>>


Mime
View raw message