Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5CF8818D76 for ; Fri, 20 Nov 2015 12:34:25 +0000 (UTC) Received: (qmail 3777 invoked by uid 500); 20 Nov 2015 12:34:25 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 3699 invoked by uid 500); 20 Nov 2015 12:34:24 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 3689 invoked by uid 99); 20 Nov 2015 12:34:24 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Nov 2015 12:34:24 +0000 Received: from mail-lf0-f54.google.com (mail-lf0-f54.google.com [209.85.215.54]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 3EC631A0044 for ; Fri, 20 Nov 2015 12:34:24 +0000 (UTC) Received: by lfaz4 with SMTP id z4so68078717lfa.0 for ; Fri, 20 Nov 2015 04:34:22 -0800 (PST) X-Received: by 10.25.141.70 with SMTP id p67mr5670060lfd.151.1448022862659; Fri, 20 Nov 2015 04:34:22 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.72.227 with HTTP; Fri, 20 Nov 2015 04:34:03 -0800 (PST) In-Reply-To: References: From: Robert Metzger Date: Fri, 20 Nov 2015 13:34:03 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Apache Flink on Hadoop YARN using a YARN Session To: "user@flink.apache.org" Content-Type: multipart/alternative; boundary=001a114037445b07030524f818d6 --001a114037445b07030524f818d6 Content-Type: text/plain; charset=UTF-8 Hi, I'll fix the link in the YARN documentation. Thank you for reporting the issue. I'm not aware of any discussions or implementations related to the scheduling. From my experience working with users and also from the mailing list, I don't think that such features are very important. Since streaming jobs usually run permanently, there is no need to queue jobs somehow. For batch jobs, YARN is taking care of the resource allocation (in practice this means that the job has to wait until the required resources are available). There are some discussions (and user requests) regarding resource elasticity going on and I think we'll add features for dynamically changing the size of a Flink cluster on YARN while a job is running. Which features are you missing wrt to scheduling in Flink? Please let me know if there is anything blocking you from using Flink in production and we'll see what we can do. Regards, Robert On Fri, Nov 20, 2015 at 1:24 PM, Ovidiu-Cristian MARCU < ovidiu-cristian.marcu@inria.fr> wrote: > Hi, > > The link to FAQ ( > https://ci.apache.org/projects/flink/flink-docs-release-0.10/faq.html) is > on the yarn setup 0.10 documentation page ( > https://ci.apache.org/projects/flink/flink-docs-release-0.10/setup/yarn_setup.html) > described in this sentence: *If you have troubles using the Flink YARN > client, have a look in the FAQ section > .* > > Is the scheduling features considered for next releases? > > Thank you. > Best regards, > Ovidiu > > On 20 Nov 2015, at 11:59, Robert Metzger wrote: > > Hi Ovidiu, > > you can submit multiple programs to a running Flink cluster (or a YARN > session). Flink does currently not have any queuing mechanism. > The JobManager will reject a program if there are not enough free > resources for it. If there are enough resources for multiple programs, > they'll run concurrently. > Note that Flink is not starting separate JVMs for the programs, so if one > program is doing a System.exit(0), it is killing the entire JVM, > including other running programs. > > You can start as many YARN sessions (or single jobs to YARN) as you have > resources available on the cluster. The resource allocation is up to the > scheduler you've configured in YARN. > > In general, we recommend to start a YARN session per program. You can also > directly submit a Flink program to YARN. > > Where did you find the link to the FAQ? The link on the front page is > working: http://flink.apache.org/faq.html > > > > On Fri, Nov 20, 2015 at 11:41 AM, Ovidiu-Cristian MARCU < > ovidiu-cristian.marcu@inria.fr> wrote: > >> Hi, >> >> I am currently interested in experimenting on Flink over Hadoop YARN. >> I am documenting from the documentation we have here: >> https://ci.apache.org/projects/flink/flink-docs-release-0.10/setup/yarn_setup.html >> >> There is a subsection *Start Flink Session* which states the following: *A >> session will start all required Flink services (JobManager and >> TaskManagers) so that you can submit programs to the cluster. Note that you >> can run multiple programs per session.* >> >> Can you be more precise regarding the multiple programs per session? If I >> submit multiple programs concurently what will happen (can I?)? Maybe they >> will run in a FIFO fashion or what should I expect? >> >> The internals section specify that users can execute multiple Flink Yarn >> sessions in parallel. This is great, this invites to static partitioning of >> resources in order to run multiple applications concurrently. Do you >> support a fair scheduler similar to what Spark claims it has? >> >> There is FAQ section ( >> https://ci.apache.org/projects/flink/flink-docs-release-0.10/faq.html) >> resource that is missing, can this be updated? >> >> Thank you. >> >> Best regards, >> Ovidiu >> >> > > > --001a114037445b07030524f818d6 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,
I'll fix the link in the YARN documentation. T= hank you for reporting the issue.

I'm not awar= e of any discussions or implementations related to the scheduling. From my = experience working with users and also from the mailing list, I don't t= hink that such features are very important.
Since streaming jobs = usually run permanently, there is no need to queue jobs somehow.
= For batch jobs, YARN is taking care of the resource allocation (in practice= this means that the job has to wait until the required resources are avail= able).

There are some discussions (and user reques= ts) regarding resource elasticity going on and I think we'll add featur= es for dynamically changing the size of a Flink cluster on YARN while a job= is running.

Which features are you missing wrt to= scheduling in Flink? Please let me know if there is anything blocking you = from using Flink in production and we'll see what we can do.
=
Regards,
Robert



On Fri, No= v 20, 2015 at 1:24 PM, Ovidiu-Cristian MARCU <ovidiu-cristia= n.marcu@inria.fr> wrote:
Hi,

The link to FAQ (<= a href=3D"https://ci.apache.org/projects/flink/flink-docs-release-0.10/faq.= html" target=3D"_blank">https://ci.apache.org/projects/flink/flink-docs-rel= ease-0.10/faq.html)=C2=A0is on the yarn setup 0.10 documentation page (= https://ci.apache.org/projects/flink/= flink-docs-release-0.10/setup/yarn_setup.html) described in this senten= ce:=C2=A0If you have troubles using the Flink YARN clien= t, have a look in the=C2=A0FAQ section.
<= span style=3D"font-size:14px;background-color:rgb(255,255,255)">
=
Is the scheduling features considered for next releases?<= /font>

Thank you.
Best regards,
Ovidiu

On 20 Nov 2015, at 11:59, Robert Metzger <rmetzger@apache.org> wrote= :

Hi=C2=A0O= vidiu,

you can submit multiple programs to a runnin= g Flink cluster (or a YARN session). Flink does currently not have any queu= ing mechanism.
The JobMan= ager will reject a program if there are not enough free resources for it. I= f there are enough resources for multiple=C2=A0programs, they'll r= un concurrently.
Note tha= t Flink is not starting separate JVMs for the=C2=A0programs, so if one= =C2=A0program=C2=A0is doing a System.exit(0), it is killing the entire= JVM, including other running=C2=A0programs.

You can start as many YARN sessions (or single jobs to YARN) as you have = resources available on the cluster. The resource allocation is up to the sc= heduler you've configured in YARN.

In gen= eral, we recommend to start a YARN session per program. You can also direct= ly submit a Flink program to YARN.

Where did = you find the link to the FAQ? The link on the front page is working:=C2=A0<= a href=3D"http://flink.apache.org/faq.html" target=3D"_blank">http://flink.= apache.org/faq.html
<= br>


On Fri, Nov 20,= 2015 at 11:41 AM, Ovidiu-Cristian MARCU <ovidiu-cristian.mar= cu@inria.fr> wrote:
Hi,

I am currently interest= ed in experimenting on Flink over Hadoop YARN.
I am documenting f= rom the documentation we have here:=C2=A0https://ci.apache.org/projects/flink/flink-docs-release-0.10/setup/yarn= _setup.html

There is a subsection Start Fli= nk Session which states the following:=C2=A0A session will start all = required Flink services (JobManager and TaskManagers) so that you can submi= t programs to the cluster. Note that you can run multiple programs per sess= ion.

Can you be more preci= se regarding the multiple programs per session? If I submit multiple progra= ms concurently what will happen (can I?)? Maybe they will run in a FIFO fas= hion or what should I expect?

The internals sectio= n specify that users can execute multiple Flink Yarn sessions in parallel. = This is great, this invites to static partitioning of resources in order to= run multiple applications concurrently. Do you support a fair scheduler si= milar to what Spark claims it has? =C2=A0

There is= FAQ section (https://ci.apache.org/projects/flink= /flink-docs-release-0.10/faq.html) resource that is missing, can this b= e updated?

Thank you.

Bes= t regards,
Ovidiu
=C2=A0
=

--001a114037445b07030524f818d6--