flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tzulitai <...@git.apache.org>
Subject [GitHub] flink pull request #5268: [FLINK-8398] [kinesis, tests] Harden KinesisDataFe...
Date Wed, 10 Jan 2018 06:14:23 GMT
GitHub user tzulitai opened a pull request:


    [FLINK-8398] [kinesis, tests] Harden KinesisDataFetcherTest unit tests

    ## What is the purpose of the change
    Prior to this PR, many of the `KinesisDataFetcherTest` unit tests relied on thread sleeps
to wait until a certain operation occurs to allow the test to pass. This test behaviour is
very flaky, and should be replaced with `OneShotLatch`.
    ## Brief change log
    - 94b4591: Several minor cleanups of confusing implementations / code smells in the `KinesisDataFetcherTest`
and related test classes. The commit message explains what exactly was changed.
    - 547d19f: Remove thread sleeps in unit tests, and replace them with `OneShotLatch`.
    ## Verifying this change
    No test coverage should have been affected by this change.
    The existing tests in `KinesisDataFetcherTest` verifies this.
    ## Does this pull request potentially affect one of the following parts:
      - Dependencies (does it add or upgrade a dependency): (yes / *no*)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes
/ *no*)
      - The serializers: (yes / *no* / don't know)
      - The runtime per-record code paths (performance sensitive): (yes / *no* / don't know)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: (yes / *no* / don't know)
      - The S3 file system connector: (yes / *no* / don't know)
    ## Documentation
      - Does this pull request introduce a new feature? (yes / *no*)
      - If yes, how is the feature documented? (*not applicable* / docs / JavaDocs / not documented)

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tzulitai/flink FLINK-8398

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5268
commit 94b45919afa5a3ec3ce68c45e57f7989397f9640
Author: Tzu-Li (Gordon) Tai <tzulitai@...>
Date:   2018-01-10T02:11:31Z

    [FLINK-8398] [kinesis, tests] Cleanup confusing implementations in KinesisDataFetcherTest
and related classes
    The previous implementation of the TestableKinesisDataFetcher was
    confusing in various ways, causing it hard to be re-used for other
    tests. This commit contains the following various cleaups:
    - Remove confusing mocks of source context and checkpoint lock. We now
      allow users of the TestableKinesisDataFetcher to provide a source
      context, which should provide the checkpoint lock.
    - Remove override of emitRecordAndUpdateState(). Strictly speaking, that
      method should be final. It was previously overriden to allow
      verifying how many records were output by the fetcher. That
      verification would be better implemented within a mock source context.
    - Properly parameterize the output type for the
    - Remove use of PowerMockito in KinesisDataFetcherTest.
    - Use CheckedThreads to properly capture any exceptions in fetcher /
      consumer threads in unit tests.
    - Use assertEquals / assertNull instead of assertTrue where-ever

commit 547d19f9196512231661f427f3792f2e1f831339
Author: Tzu-Li (Gordon) Tai <tzulitai@...>
Date:   2018-01-10T05:41:49Z

    [FLINK-8398] [kinesis, tests] Stabilize flaky KinesisDataFetcherTests
    Prior to this commit, several unit tests in KinesisDataFetcherTest
    relied on sleeps to wait until a certain operation happens, in order for
    the test to pass.
    This commit removes those sleeps and replaces the test behaviours with



View raw message