flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Examples of Flink windowing unit tests, and option to bring in flink-spector to Flink
Date Wed, 03 May 2017 17:39:37 GMT
We should definitely wait wait Alexander has to say, he’s the original creator of flink-spectir.
I’m cc’ing him but I’m not sure whether that’s still his current address.

> On 3. May 2017, at 19:14, Bowen Li <bowen.li@offerupnow.com> wrote:
> Hi Aljoscha,
>    My team is looking at forking flink-spector and make it compatible with
> Flink 1.2. Thanks for confirming that we are in the right direction.
>    What're the barriers of bring in flink-spector to Apache Flink?
> flink-spector is of Apache License 2.0, the same as Apache Flink. The only
> potential issue we see right now is the copyright header they add to each
> file, which looks like below. Do you mind have it stay there, or should we
> try to get their approval and remove the copyright header when merging?
> ```
> /*
> * Copyright 2015 Otto (GmbH & Co KG)
> *
> * Licensed under the Apache License, Version 2.0 (the "License");
> * you may not use this file except in compliance with the License.
> * You may obtain a copy of the License at
> *
> * http://www.apache.org/licenses/LICENSE-2.0
> *
> * Unless required by applicable law or agreed to in writing, software
> * distributed under the License is distributed on an "AS IS" BASIS,
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> * See the License for the specific language governing permissions and
> * limitations under the License.
> */
> ```
> On Wed, May 3, 2017 at 3:20 AM, Aljoscha Krettek <aljoscha@apache.org>
> wrote:
>> Hi,
>> Yes, unfortunately there are no good unit test examples for testing Flink
>> jobs (with windowing or without). And yes, I would like to have something
>> like flink-spector (or flink-spector) as part of Flink as the promoted way
>> of doing Job testing. In my opinion this is blocked on a good way of
>> verifying that assertions succeed inside operators and on an API to
>> programmatically monitor and cancel a running streaming Job.
>> If you had those two components (asynchronous job management API and
>> assertion verification) you could do something similar to PAssert (
>> https://beam.apache.org/documentation/sdks/javadoc/0.
>> 6.0/index.html?org/apache/beam/sdk/testing/PAssert.html <
>> https://beam.apache.org/documentation/sdks/javadoc/0.
>> 6.0/index.html?org/apache/beam/sdk/testing/PAssert.html>) in Apache Beam
>> that is used for a lot of tests. The idea is that assertion operators
>> inside the pipeline wait for a condition to be true (or fail of some
>> negative condition is violated) and signal success by updating a unique
>> Accumulator value. The test would use the asynchronous job API to listen on
>> Accumulator updates and wait until all required conditions have been met.
>> Then it would stop the job and the test would succeed.
>> Best,
>> Aljoscha
>>> On 2. May 2017, at 22:01, Bowen Li <bowen.li@offerupnow.com> wrote:
>>> Hi Flink folks!
>>>    Are there any concrete good examples of unit testing Flink windowing?
>>> I've been searching around, and all those available online are quite
>>> ambiguous.
>>>    Another thing is, I found the flink-spector
>>> <https://github.com/ottogroup/flink-spector> library is pretty awesome,
>>> simple to understand and use. But it's kind of out of maintenance, and
>> only
>>> supports 1.4 now. Is it possible to bring flink-spector into Apache
>> Flink,
>>> so we can better upgrade and maintain it?
>>> Thank you!
>>> Bowen

View raw message