flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Metzger (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-4114) Need a way to manage multiple named, long-lived jobs on a single YARN cluster in an automated manner
Date Tue, 28 Jun 2016 09:31:57 GMT

     [ https://issues.apache.org/jira/browse/FLINK-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Robert Metzger updated FLINK-4114:
----------------------------------
    Component/s: YARN Client
     Issue Type: Improvement  (was: Bug)

> Need a way to manage multiple named, long-lived jobs on a single YARN cluster in an automated
manner
> ----------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-4114
>                 URL: https://issues.apache.org/jira/browse/FLINK-4114
>             Project: Flink
>          Issue Type: Improvement
>          Components: YARN Client
>            Reporter: Chris Hogue
>
> We are running several Flink jobs on a single YARN cluster. Currently each Flink job
is run in its own YARN session (and thus its own YARN application ID). The difficulty comes
in that we want to manage each of these jobs individually by name. For example we want to
start, stop, update one job without affecting others. The primary access to these jobs is
via the YARN application ID, which is not meaningful to discern which flink job it is running.
> It would be nice if we had tools that would allow us to manage the flink jobs by name
and have it do the right thing with the YARN session. Today we can use 'flink run' and have
it start a YARN session for that job, but from that point forward we have only the YARN application
ID to work with.
> As a concrete example suppose we have 2 jobs with names JobA and JobB. We'd want a way
to so something like: 
> flink run <JobA jar>; flink run <JobB jar>
> We'd then want to be able to call:
> flink cancel JobA
> The cancel command would spin down the YARN session for JobA in addition to the flink
job, leaving JobB running as normal. I've simplified the commands leaving out other options
for illustrative purposes. And we'll want to be able to use savepoints through these steps
as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message