flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Apache Flink on Hadoop YARN using a YARN Session
Date Fri, 20 Nov 2015 10:59:59 GMT
Hi Ovidiu,

you can submit multiple programs to a running Flink cluster (or a YARN
session). Flink does currently not have any queuing mechanism.
The JobManager will reject a program if there are not enough free resources
for it. If there are enough resources for multiple programs, they'll run
concurrently.
Note that Flink is not starting separate JVMs for the programs, so if one
program is doing a System.exit(0), it is killing the entire JVM, including
other running programs.

You can start as many YARN sessions (or single jobs to YARN) as you have
resources available on the cluster. The resource allocation is up to the
scheduler you've configured in YARN.

In general, we recommend to start a YARN session per program. You can also
directly submit a Flink program to YARN.

Where did you find the link to the FAQ? The link on the front page is
working: http://flink.apache.org/faq.html



On Fri, Nov 20, 2015 at 11:41 AM, Ovidiu-Cristian MARCU <
ovidiu-cristian.marcu@inria.fr> wrote:

> Hi,
>
> I am currently interested in experimenting on Flink over Hadoop YARN.
> I am documenting from the documentation we have here:
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/setup/yarn_setup.html
>
> There is a subsection *Start Flink Session* which states the following: *A
> session will start all required Flink services (JobManager and
> TaskManagers) so that you can submit programs to the cluster. Note that you
> can run multiple programs per session.*
>
> Can you be more precise regarding the multiple programs per session? If I
> submit multiple programs concurently what will happen (can I?)? Maybe they
> will run in a FIFO fashion or what should I expect?
>
> The internals section specify that users can execute multiple Flink Yarn
> sessions in parallel. This is great, this invites to static partitioning of
> resources in order to run multiple applications concurrently. Do you
> support a fair scheduler similar to what Spark claims it has?
>
> There is FAQ section (
> https://ci.apache.org/projects/flink/flink-docs-release-0.10/faq.html)
> resource that is missing, can this be updated?
>
> Thank you.
>
> Best regards,
> Ovidiu
>
>

Mime
View raw message