flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8656) Add CLI command for rescaling
Date Wed, 14 Feb 2018 14:12:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364090#comment-16364090
] 

ASF GitHub Bot commented on FLINK-8656:
---------------------------------------

GitHub user tillrohrmann opened a pull request:

    https://github.com/apache/flink/pull/5487

    [FLINK-8656] [flip6] Add modify CLI command to rescale Flink jobs

    ## What is the purpose of the change
    
    Jobs can now be rescaled by calling flink modify <JOB_ID> -p <PARALLELISM>.
    Internally, the CliFrontend will send the corresponding REST call and poll
    for status updates.
    
    This PR is based on #5454.
    
    ## Brief change log
    
    - Add `modify` call to `CliFrontend`
    - Add `ClusterClient#rescaleJob` method with default implementation
    - Implement `RestClusterClient#rescalJob` method to trigger asynchronous rescale operation
via REST and poll for its status updates
    
    ## Verifying this change
    
    - Tested manually
    - Added `CliFrontendModifyTest`
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: (no)
      - The S3 file system connector: (no)
    
    ## Documentation
    
      - Does this pull request introduce a new feature? (yes)
      - If yes, how is the feature documented? (JavaDocs + stdout help)


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tillrohrmann/flink rescaleCommand

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5487.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5487
    
----
commit d9159228091cae9ebbd1bb718b69e6cf452881e1
Author: Till Rohrmann <trohrmann@...>
Date:   2018-02-13T11:41:44Z

    [FLINK-8643] [flip6] Use JobManagerOptions#SLOT_REQUEST_TIMEOUT in ExecutionGraph
    
    This commit changes the initialization of the ExecutionGraph to use the
    JobManagerOptions#SLOT_REQUEST_TIMEOUT for the slot allocation. Furthermore,
    it changes the behaviour of the SlotPool#ProviderAndOwner implementation such
    that the timeout is given to it via the SlotProvider#allocateSlot call.

commit 19780c9d284914ec51e92231536315299a3c2da3
Author: Till Rohrmann <trohrmann@...>
Date:   2018-02-13T12:18:01Z

    [hotfix] [flip6] Remove unnecessary timeout from SlotPool

commit 9924776c92a378cef144c0767f1ff18b799d52e9
Author: Till Rohrmann <trohrmann@...>
Date:   2018-02-13T14:33:11Z

    [FLINK-8647] [flip6] Introduce JobMasterConfiguration
    
    This commit introduces a JobMasterConfiguration which contains JobMaster specific
    configuration settings.

commit fde75841de2e27cb7380f3a28066a99e2c1a690d
Author: zentol <chesnay@...>
Date:   2018-01-23T12:50:32Z

    [FLINK-8475][config][docs] Integrate HA-ZK options
    
    This closes #5462.

commit 788a17fdbd4aaf3429ead4491ede197fc775b1f0
Author: zentol <chesnay@...>
Date:   2018-01-23T13:04:36Z

    [FLINK-8475][config][docs] Integrate YARN options
    
    This closes #5463.

commit fcd783358c282e61bf12e0c18298c237c85a6695
Author: Till Rohrmann <trohrmann@...>
Date:   2018-02-13T15:08:38Z

    [hotfix] [tests] Simplify JobMasterTest

commit 8206e6f13809c0b60bfaf776bc386088f535e723
Author: Till Rohrmann <trohrmann@...>
Date:   2018-02-13T15:10:09Z

    [FLINK-8546] [flip6] Respect savepoints and restore from latest checkpoints
    
    Let the JobMaster respect checkpoints and savepoints. The JobMaster will always
    try to restore the latest checkpoint if there is one available. Next it will check
    whether savepoint restore settings have been set. If so, then it will try to restore
    the savepoint. Only if these settings are not set, the job will be started from
    scratch.

commit 057a95b7328b1cca7b78bf1dd25e8d048df70410
Author: Till Rohrmann <trohrmann@...>
Date:   2018-02-13T15:11:37Z

    [hotfix] Fix checkstyle violations in ExecutionGraph

commit 9930b0991320bcff268ca82db6378df8976560dc
Author: Till Rohrmann <trohrmann@...>
Date:   2018-02-13T15:12:41Z

    [FLINK-8627] Introduce new JobStatus#SUSPENDING to ExecutionGraph
    
    The new JobStatus#SUSPENDING says that an ExecutionGraph has been suspended but its
    clean up has not been done yet. Only after all Executions have been canceled, the
    ExecutionGraph will enter the SUSPENDED state and complete the termination future
    accordingly.

commit 6c51ad306c90464572353168ecafdb962794747e
Author: Till Rohrmann <trohrmann@...>
Date:   2018-02-13T15:14:41Z

    [FLINK-8629] [flip6] Allow JobMaster to rescale jobs
    
    This commit adds the functionality to rescale a job or parts of it to
    the JobMaster. In order to rescale a job, the JobMaster does the following:
    1. Take a savepoint
    2. Create a rescaled ExecutionGraph from the JobGraph
    3. Initialize it with the taken savepoint
    4. Suspend the old ExecutionGraph
    5. Restart the new ExecutionGraph once the old ExecutionGraph has been suspended

commit 990aa95040264ffd3c3ab20638cb8212af68d155
Author: Till Rohrmann <trohrmann@...>
Date:   2018-02-13T15:34:31Z

    [FLINK-8633] [flip6] Expose rescaling of jobs via the Dispatcher
    
    This commit exposes the JobMaster#rescaleJob via the Dispatcher. This will
    allow it to call this functionality from a REST handler.

commit ca85f392c0caea751bf7805beb4428eba5cf851f
Author: Till Rohrmann <trohrmann@...>
Date:   2018-02-03T16:41:51Z

    [FLINK-8631] [rest] Add support for generic types to the RestClient
    
    This commit allows the Restclient to receive generic response types. In order
    to do this, the MessageHeaders contain now information about the generic
    type parameters of the response type.

commit 381099612b78ad1ab7bc5f239d2fd7c618c74222
Author: Till Rohrmann <trohrmann@...>
Date:   2018-02-02T10:06:35Z

    [FLINK-8634] [rest] Introduce job rescaling REST handler
    
    Add rescaling REST handler as a sub class of the
    AbstractAsynchronousOperationHandlers.

commit f4e9f71b4821dabfadc47491e3b4a67ef5e3dab3
Author: Till Rohrmann <trohrmann@...>
Date:   2018-02-02T10:23:00Z

    [FLINK-8632] [flip6] Introduce generalized asynchronous operation handlers
    
    The asynchronous operation handlers are the generalization of the SavepointHandlers.
    They consist of a Trigger- and a StatusHandler. The TriggerHandler is used to trigger
    an asynchronous operation. The handler stores the operation future and returns a
    trigger id. The trigger id can be used to query the status of the operation via the
    StatusHandler. Once the operation has completed, the StatusHandler will return the
    result.

commit 79f20916a1366fed9caee27c6fcfe932d99c7482
Author: Till Rohrmann <trohrmann@...>
Date:   2018-02-11T18:50:46Z

    [FLINK-8635] [rest] Register rescaling handlers at web endpoint

commit 7081cba4cd3ba377b64a4d165e1fc888be99873f
Author: Till Rohrmann <trohrmann@...>
Date:   2018-02-13T16:29:32Z

    [FLINK-8656] [flip6] Add modify CLI command to rescale Flink jobs
    
    Jobs can now be rescaled by calling flink modify <JOB_ID> -p <PARALLELISM>.
    Internally, the CliFrontend will send the corresponding REST call and poll
    for status updates.

----


> Add CLI command for rescaling
> -----------------------------
>
>                 Key: FLINK-8656
>                 URL: https://issues.apache.org/jira/browse/FLINK-8656
>             Project: Flink
>          Issue Type: New Feature
>          Components: Client
>    Affects Versions: 1.5.0
>            Reporter: Till Rohrmann
>            Assignee: Till Rohrmann
>            Priority: Major
>              Labels: flip-6
>             Fix For: 1.5.0
>
>
> The REST rescaling calls should be made accessible via the {{CliFrontend}}. In order
to do that I propose to add a {{modify}} command to the {{CliFrontend}} to which we can pass
a new parallelism.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message