flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7918) Run AbstractTestBase tests on Flip-6 MiniCluster
Date Tue, 28 Nov 2017 16:56:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269039#comment-16269039

ASF GitHub Bot commented on FLINK-7918:

GitHub user tillrohrmann opened a pull request:


    [FLINK-7918] Run AbstractTestBase tests on Flip-6 MiniCluster

    ## What is the purpose of the change
    Extend `MiniClusterResource` to instantiate a Flip-6 `MiniCluster` if the system property
`codebase` has been set to `flip6`. This allows to run all tests which are based on `AbstractTestBase`
on the `Flip-6` code base.
    ## Brief change log
    - Use system property `codebase` to distinguish between Flip-6 and old code base
    - Instantiate respective `MiniCluster` in `MiniClusterResource` depending on the system
    ## Does this pull request potentially affect one of the following parts:
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: (no)
      - The S3 file system connector: (no)
    ## Documentation
      - Does this pull request introduce a new feature? (no)
      - If yes, how is the feature documented? (not applicable)

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tillrohrmann/flink flip6TestBase

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5095
commit 1f6cb24a2b8e4b062fe12d756bfd5950ab35b0ce
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-11-14T22:50:52Z

    [FLINK-8078] Introduce LogicalSlot interface
    The LogicalSlot interface decouples the task deployment from the actual
    slot implementation which at the moment is Slot, SimpleSlot and SharedSlot.
    This is a helpful step to introduce a different slot implementation for

commit aff11aaf02c7b5b667780031e01a1b08d85e036b
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-11-15T13:20:27Z

    [FLINK-8085] Thin out LogicalSlot interface
    Remove isCanceled, isReleased method and decouple logical slot from Execution by
    introducing a Payload interface which is set for a LogicalSlot. The Payload interface
    is implemented by the Execution and allows to fail an implementation and obtaining
    a termination future.
    Introduce proper Execution#releaseFuture which is completed once the Execution's
    assigned resource has been released.

commit 040cb326086968b5fe5d66dcc82835ee9dd092dc
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-11-24T17:03:49Z

    [FLINK-8087] Decouple Slot from AllocatedSlot
    This commit introduces the SlotContext which is an abstraction for the SimpleSlot
    to obtain the relevant slot information to do the communication with the
    TaskManager without relying on the AllocatedSlot which is now only used by the

commit 5d7e117392a51a485a5cb682c4a2c12028e73b97
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-11-24T17:06:10Z

    [FLINK-8088] Associate logical slots with the slot request id
    Before logical slots like the SimpleSlot and SharedSlot where associated to the
    actually allocated slot via the AllocationID. This, however, was sub-optimal because
    allocated slots can be re-used to fulfill also other slot requests (logical slots).
    Therefore, we should bind the logical slots to the right id with the right lifecycle
    which is the slot request id.

commit 75f49d4bb3f38c7bfaee101607a459d558c00ba8
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-11-13T14:42:07Z

    [FLINK-8089] Also check for other pending slot requests in offerSlot
    Not only check for a slot request with the right allocation id but also check
    whether we can fulfill other pending slot requests with an unclaimed offered
    slot before adding it to the list of available slots.

commit b53729e74ccc31c634c5ac7db5e37ca45a66db8d
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-11-24T17:08:38Z

    [FLINK-7956] [flip6] Add support for queued scheduling with slot sharing to SlotPool
    This commit adds support for queued scheduling with slot sharing to the
    SlotPool. The idea of slot sharing is that multiple tasks can run in the
    same slot. Moreover, queued scheduling means that a slot request must not
    be completed right away but at a later point in time. This allows to
    start new TaskExecutors in case that there are no more slots left.
    The main component responsible for the management of shared slots is the
    SlotSharingManager. The SlotSharingManager maintains internally a tree-like
    structure which stores the SlotContext future of the underlying
    AllocatedSlot. Whenever this future is completed potentially pending
    LogicalSlot instantiations are executed and sent to the slot requester.
    A shared slot is represented by a MultiTaskSlot which can harbour multiple
    TaskSlots. A TaskSlot can either be a MultiTaskSlot or a SingleTaskSlot.
    In order to represent co-location constraints, we first obtain a root
    MultiTaskSlot and then allocate a nested MultiTaskSlot in which the
    co-located tasks are allocated. The corresponding SlotRequestID is assigned
    to the CoLocationConstraint in order to make the TaskSlot retrievable for
    other tasks assigned to the same CoLocationConstraint.
    Port SchedulerSlotSharingTest, SchedulerIsolatedTasksTest and
    ScheduleWithCoLocationHintTest to run with SlotPool.
    Restructure SlotPool components.
    Add SlotSharingManagerTest, SlotPoolSlotSharingTest and

commit b1d520efbe258070d92e9b66904e4e1d3d31fee6
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-11-24T22:29:53Z

    [hotfix] Speed up RecoveryITCase

commit 5df0b85ed39edb35e22ae684306c6bc564ff02e4
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-23T14:21:14Z

    [FLINK-7903] [tests] Add flip6 build profile
    The flip6 build profile only runs the Flip-6 related test cases. Moreover,
    all Flip-6 related test cases are excluded when not running the flip6 build
    profile. This should reduce testing time when adding more and more Flip-6
    test cases.

commit df818af4a5b33cb9f9c8c1606aa715b08b926093
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-24T10:34:53Z

    Include flink-test-utils-junit in all submodules to make the Category marker interfaces
Flip6 and OldAndFlip6 available

commit ea8ac2f75ab176a9b6fd6491a289ef3f7433aff6
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-11-27T08:29:54Z

    [hotfix] [tests] Speed up queryable state IT tests by removing sleep

commit 8344dde82f5c8a4307084f5a0558c4520a5cd880
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-23T14:24:58Z

    [FLINK-7904] Enable Flip6 build profile on Travis
    This adds a new Travis build matrix entry which runs the Flip-6 build profile

commit 8bb13fd9cbea940e98552367128b9dde70ab80d0
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-24T13:38:02Z

    Add core+tests TEST option

commit 53c15ce1d9deb2a8060764da75365df052ad7ed9
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-24T13:51:27Z

    [hotfix] Remove unnecessary surefire plugin version in flink-connector-elasticsearch5

commit 16b5caab9422ec1d60192bd1adba2c725ad0bab0
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-25T10:03:31Z

    Reuse caches and split core+test into core and tests Travis build

commit 76fe0d635c6a0a41c47317bd1dd9c7e887886ca2
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-24T09:32:05Z

    [FLINK-7909] Unify Flink test bases
    Introduce a MiniClusterResource which is used by the AbstractTestBase to start
    and shut down a FlinkMiniCluster. Additionally, this resource registers the proper
    Stream- and ExecutionEnvironment which is now the only way for tests to start
    jobs. This change will thus allow to centrally control which FlinkCluster will
    be started for all test bases.

commit 2a5015d985e8f40695b4a6676772f32c0a7602f3
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-24T16:53:40Z

    [FLINK-7910] [tests] Generalize Test(Stream)Environment to use JobExecutor
    This commit introduces the JobExecutor interface which abstracts the actual mini cluster
    from the Test(Stream)Environment. By letting the Flip-6 MiniCluster as well as the
    FlinkMiniCluster implement this interface, we can run all test base jobs either on the
    Flip-6 mini cluster or on the current mini cluster.

commit 3395d3041caedd754e2414c8d03b4f3d47f2ba59
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-25T11:29:38Z

    Fix travis.yml

commit a2f9734463df2eafd05f48e89563611adbd549f3
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-24T14:20:15Z

    [FLINK-7909] Replace StreamingMultipleProgramsTestBase by AbstractTestBase
    The AbstractTestBase fully subsumes the functionality of the
    StreamingMultipleProgramsTestBase since it now is the most general test base
    for streaming and batch jobs. As a consequence, we can safely remove the
    StreamingMultipleProgramsTestBase and let all corresponding tests extend from

commit 87a188dab5af31d35251fd644bda63040cc559f0
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-25T09:43:06Z

    Fix failing HadoopCompatibility test cases

commit 9fd8761393c9fb17094eb95f9603a2cc58855e66
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-25T10:08:29Z

    Fix flink-connector-filesystem

commit 7d9c1f285866d54076f877419ae1c7ab9ecd1ca5
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-25T12:55:06Z

    Fix failing test cases

commit 0be28d9cb940488dc204bb058bda2c6c04d619bf
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-25T13:13:39Z


commit d8c8c8e9cf9634f66da33100d3ef473886ef4f81
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-10-23T15:02:58Z

    [FLINK-7918] Run AbstractTestBase tests on Flip-6 MiniCluster


> Run AbstractTestBase tests on Flip-6 MiniCluster
> ------------------------------------------------
>                 Key: FLINK-7918
>                 URL: https://issues.apache.org/jira/browse/FLINK-7918
>             Project: Flink
>          Issue Type: New Feature
>          Components: Tests
>    Affects Versions: 1.4.0
>            Reporter: Till Rohrmann
>            Assignee: Till Rohrmann
>              Labels: flip-6
> In order to extend our test coverage for Flip-6, we should enable all {{AbstractTestBase}}
tests to be executed via Flip-6's {{MiniCluster}}.

This message was sent by Atlassian JIRA

View raw message