flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin Williams <colin.williams.seat...@gmail.com>
Subject Re: Writing an Integration test for flink-metrics
Date Tue, 24 Oct 2017 05:43:32 GMT
Thanks for the help. I ended up creating a custom metric reporter and
accessing it's fields in an integration test. However I do think that the
test that Martin checked in is another good way to test. I opened
https://issues.apache.org/jira/browse/FLINK-7907 regarding the missing
Scala examples in the metrics documentation. I can take a stab at that on
the weekend if somebody doesn't beat me to it.

On Fri, Oct 13, 2017 at 6:09 AM, Martin Eden <martineden131@gmail.com>
wrote:

> Hi,
> Not merged in yet but this is an example pr that is mocking metrics and
> checking they are properly updated:
> https://github.com/apache/flink/pull/4725
>
>
> On Fri, Oct 13, 2017 at 1:49 PM, Aljoscha Krettek <aljoscha@apache.org>
> wrote:
>
>> I think we could add this functionality to the (operator) test harnesses.
>> I.e. add a mock MetricGroup thingy in there that you can query to check the
>> state of metrics.
>>
>>
>> On 13. Oct 2017, at 13:50, Chesnay Schepler <chesnay@apache.org> wrote:
>>
>> I meant that you could unit-test the behavior of the function in
>> isolation. You could create a dummy metric group that
>> verifies that the correct counters are being registered (based on names i
>> guess), as well as provide access to them.
>> Mock some input and observe whether the counter value is being modified.
>>
>> Whether this is a viable option depends a bit on the complexity of the
>> function of course, that is how much how mocking
>> you would have to do.
>>
>> On 13.10.2017 11:18, Piotr Nowojski wrote:
>>
>> For testing Link applications in general you can read
>> https://ci.apache.org/projects/flink/flink-docs-release
>> -1.4/dev/stream/testing.html
>>
>> However as we said before, testing metrics would require using custom or
>> a imx reporter.
>>
>> Yes, please report this bug in Jira.
>>
>> Thanks, Piotrek
>>
>> On 13 Oct 2017, at 04:31, Colin Williams <colin.williams.seattle@gmail.
>> com> wrote:
>>
>> Team wants an integration test, I'm not sure what unit test you had in
>> mind. Actually feel that I've been trying to avoid the reporter method but
>> that would be more end to end.
>>
>> The documentation for metrics and Scala are missing with the exception of
>> Gauge: https://ci.apache.org/projects/flink/flink-docs-release-1.3/
>> monitoring/metrics.html . Should I file a issue against that?
>>
>> Then it leaves you guessing a little bit how to implement Counters. One
>> approach tried was using objects
>>
>> object PointFilter extends RichMapFunction[...
>>
>>   @transient lazy val someCounter = getRuntimeContext.getMetricGroup.counter(...)
>>
>>
>> This allowed access to the counter before and after execution . However
>> between the unit tests the Counter kept its value also and that's a no for
>> the test. Think that might be an issue with ScalaTest.
>>
>> I've tried to get at the counter from some other directions like trying
>> to find a way to inject a reporter to get it's state. But don't see a way
>> to do it. So probably the best thing to do is fire up something to collect
>> the metrics from the reporter.
>>
>> On Thu, Oct 12, 2017 at 5:29 AM, Chesnay Schepler <chesnay@apache.org>
>> wrote:
>>
>>> Well damn, i should've read the second part of the initial mail.
>>>
>>> I'm wondering though, could you not unit-test this behavior?
>>>
>>>
>>> On 12.10.2017 14:25, Chesnay Schepler wrote:
>>>
>>>> You could also write a custom reporter that opens a socket or similar
>>>> for communication purposes.
>>>>
>>>> You can then either query it for the metrics, or even just trigger the
>>>> verification in the reporter,
>>>> and fail with an error if the reporter returns an error.
>>>>
>>>> On 12.10.2017 14:02, Piotr Nowojski wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Doing as you proposed using JMXReporter (or custom reporter) should
>>>>> work. I think there is no easier way to do this at the moment.
>>>>>
>>>>> Piotrek
>>>>>
>>>>> On 12 Oct 2017, at 04:58, Colin Williams <
>>>>>> colin.williams.seattle@gmail.com> wrote:
>>>>>>
>>>>>> I have a RichMapFunction and I'd like to ensure Meter fields are
>>>>>> properly incremented. I've been trying to think of the best way to
do this.
>>>>>> Currently I think that I'd need to either implement my own reporter
(or use
>>>>>> JMX) and write to a socket, create a listener and wait for the reporter
to
>>>>>> send the message.
>>>>>>
>>>>>> Is this a good approach for writing the test, or should I be
>>>>>> considering something else?
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
>>
>

Mime
View raw message