flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8306) FlinkKafkaConsumerBaseTest has invalid mocks on final methods
Date Fri, 12 Jan 2018 02:15:01 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323429#comment-16323429

ASF GitHub Bot commented on FLINK-8306:

GitHub user tzulitai opened a pull request:


    [FLINK-8306] [kafka, tests] Fix invalid mock verifications on final method

    ## What is the purpose of the change
    This is a reworked version of #5200.
    Instead of introducing a new interface while we are still unsure of how the refactoring
of the Kafka consumer should head towards, this version is a simple fix to have proper mocks
in the `FlinkKafkaConsumerBase`.
    Only the last two commits are relevant (PR is based on #5188).
    ## Brief change log
    - ca1aa00 Remove invalid mock verifications, and introduce a proper `MockFetcher` mock.
    - d08727c is a hotfix that corrects a stale comment on a no-longer existing behaviour
    ## Verifying this change
    This change is a code cleanup, and is already covered by existing tests.
    ## Does this pull request potentially affect one of the following parts:
      - Dependencies (does it add or upgrade a dependency): (yes / **no**)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes
/ **no**)
      - The serializers: (yes / **no** / don't know)
      - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
      - The S3 file system connector: (yes / **no** / don't know)
    ## Documentation
      - Does this pull request introduce a new feature? (yes / **no**)
      - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tzulitai/flink FLINK-8306-reworked

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5284
commit 735972332ff25623898b25f210b7277aafbc7351
Author: Tzu-Li (Gordon) Tai <tzulitai@...>
Date:   2017-12-20T00:10:44Z

    [FLINK-8296] [kafka] Rework FlinkKafkaConsumerBaseTest to not rely on Java reflection
    Reflection was mainly used to inject mocks into private fields of the
    FlinkKafkaConsumerBase, without the need to fully execute all operator
    life cycle methods. This, however, caused the unit tests to be too
    This commit reworks the FlinkKafkaConsumerBaseTest to remove test
    consumer instantiation methods that rely on reflection for dependency
    injection. All tests now instantiate dummy test consumers normally, and
    let all tests properly execute all operator life cycle methods
    regardless of the tested logic.

commit ca1aa0074c0ae4ff8e2da4f8d87b8378c2413ef5
Author: Tzu-Li (Gordon) Tai <tzulitai@...>
Date:   2018-01-12T00:45:32Z

    [FLINK-8306] [kafka, tests] Fix mock verifications on final method
    Previously, offset commit behavioural tests relied on verifying on
    AbstractFetcher::commitInternalOffsetsToKafka(). That method is actually
    final, and could not be mocked.
    This commit fixes that by implementing a proper mock AbstractFetcher,
    which keeps track of the offset commits that go through.

commit d08727c4f8816e408dabc12a673ad24593b27023
Author: Tzu-Li (Gordon) Tai <tzulitai@...>
Date:   2017-12-20T19:54:40Z

    [hotfix] [kafka] Remove stale comment on publishing procedures of AbstractFetcher
    The previous comment mentioned "only now will the fetcher return at
    least the restored offsets when calling snapshotCurrentState()". This is
    a remnant of the previous fetcher initialization behaviour, where in the
    past the fetcher wasn't directly seeded with restored offsets on
    Since this is no longer true, this commit fixes the stale comment to
    avoid confusion.


> FlinkKafkaConsumerBaseTest has invalid mocks on final methods
> -------------------------------------------------------------
>                 Key: FLINK-8306
>                 URL: https://issues.apache.org/jira/browse/FLINK-8306
>             Project: Flink
>          Issue Type: Bug
>          Components: Kafka Connector, Tests
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>            Priority: Critical
>             Fix For: 1.5.0, 1.4.1
> The {{FlinkKafkaConsumerBaseTest}} has invalid mocks on a final {{AbstractFetcher::commitInternalOffsetsToKafka(...)}}
method. While an easy fix would be to simply make that method non-final, that is not ideal
since it would be best that the method is left final to prevent overrides in subclasses.
> This suggests that offset committing functionality is too tightly coupled with the {{AbstractFetcher}},
making it hard to perform concise tests to verify offset committing.
> I suggest that we decouple record fetching and offset committing as separate services
behind different interfaces. We should introduce a new interface, say {{KafkaOffsetCommitter}},
and test against that instead. Initially, we can simply let {{AbstractFetcher}} implement

This message was sent by Atlassian JIRA

View raw message