Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7A1F9100C3 for ; Wed, 26 Aug 2015 13:26:23 +0000 (UTC) Received: (qmail 21555 invoked by uid 500); 26 Aug 2015 13:26:23 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 21479 invoked by uid 500); 26 Aug 2015 13:26:23 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 21470 invoked by uid 99); 26 Aug 2015 13:26:23 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2015 13:26:23 +0000 Received: from mail-la0-f52.google.com (mail-la0-f52.google.com [209.85.215.52]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 676CD1A0175 for ; Wed, 26 Aug 2015 13:26:22 +0000 (UTC) Received: by labia3 with SMTP id ia3so54739293lab.3 for ; Wed, 26 Aug 2015 06:26:20 -0700 (PDT) X-Received: by 10.112.170.129 with SMTP id am1mr31192603lbc.116.1440595580888; Wed, 26 Aug 2015 06:26:20 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Aljoscha Krettek Date: Wed, 26 Aug 2015 13:26:11 +0000 Message-ID: Subject: Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails. To: "user@flink.apache.org" Content-Type: multipart/alternative; boundary=001a11c372eadd309f051e36cb8a --001a11c372eadd309f051e36cb8a Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Arnaud, I think my answer to Gwenha=C3=ABl could also be helpful to you: are you using the one-yarn-cluster-per-job mode of Flink? I.e., you are starting your Flink job with (from the doc): flink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096 ./examples/flink-java-examples-0.10-SNAPSHOT-WordCount.jar If you are, then this is almost possible on the current version of Flink. What you have to do is copy the conf directory of Flink to a separate directory that is specific to your job. There you make your modifications to the log configuration etc. Then, when you start your job you do this instead: export FLINK_CONF_DIR=3D/path/to/my/conf flink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096 ./examples/flink-java-examples-0.10-SNAPSHOT-WordCount.jar You can easily put this into your startup script, of course. I said almost possible because this requires a small fix in bin/flink. Around line 130 this line: FLINK_CONF_DIR=3D$FLINK_ROOT_DIR_MANGLED/conf needs to be replaced by this line: if [ -z "$FLINK_CONF_DIR" ]; then FLINK_CONF_DIR=3D$FLINK_ROOT_DIR_MANGLED/conf; fi (We will fix this in the upcoming version and the 0.9.1 bugfix release.) Does this help? Let us know if you are not using the one-yarn-cluster-per-job mode, then we'll have to try to find another solution. Forget the stuff about one-yarn-cluster-per-job but the other stuff should be right what you want. I hope it helps. Best, Aljoscha On Wed, 26 Aug 2015 at 15:19 LINZ, Arnaud wrote: > Ooops=E2=80=A6 Seems it was rather a write problem on the conf dir=E2=80= =A6 > > Sorry, it works! > > > > BTW, it=E2=80=99s not really nice to have an application write the config= uration > dir ; it=E2=80=99s often a root protected directory in usr/lib/flink. Is = there a > parameter to put that file elsewhere ? > > > > > > *De :* Robert Metzger [mailto:rmetzger@apache.org] > *Envoy=C3=A9 :* mercredi 26 ao=C3=BBt 2015 14:42 > *=C3=80 :* user@flink.apache.org > *Objet :* Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session > -nm), flink run without -m fails. > > > > Hi Arnaud, > > > > usually, you don't have to manually specify the JobManager address > manually with the -m argument, because it is reading it from the > conf/.yarn-session.properties file. > > > > Give me a few minutes to reproduce the issue. > > > > On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud > wrote: > > Hi, > Using last nightly build, it seems that if you call yarn-session.sh with > -nm option to give a nice application name, then you cannot submit a job > with flink run without specify the ever changing -m address > since it does not find it any longer. > > Regards, > > Arnaud > > > ------------------------------ > > > L'int=C3=A9grit=C3=A9 de ce message n'=C3=A9tant pas assur=C3=A9e sur int= ernet, la soci=C3=A9t=C3=A9 > exp=C3=A9ditrice ne peut =C3=AAtre tenue responsable de son contenu ni de= ses pi=C3=A8ces > jointes. Toute utilisation ou diffusion non autoris=C3=A9e est interdite.= Si > vous n'=C3=AAtes pas destinataire de ce message, merci de le d=C3=A9truir= e et > d'avertir l'exp=C3=A9diteur. > > The integrity of this message cannot be guaranteed on the Internet. The > company that sent this message cannot therefore be held liable for its > content nor attachments. Any unauthorized use or dissemination is > prohibited. If you are not the intended recipient of this message, then > please delete it and notify the sender. > > > --001a11c372eadd309f051e36cb8a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Arnaud,
I think my answer to=C2=A0Gwenha=C3=ABl could also be helpful to you:

are you using the one-yarn-cluster-per-job mode of Flink? I= .e., you are starting your Flink job with (from the doc):

flink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096 ./examples/flin= k-java-examples-0.10-SNAPSHOT-WordCount.jar

If= you are, then this is almost possible on the current version of Flink. Wha= t you have to do is copy the conf directory of Flink to a separate director= y that is specific to your job. There you make your modifications to the lo= g configuration etc. Then, when you start your job you do this instead:

export FLINK_CONF_DIR=3D/path/to/my/conf
fl= ink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096 ./examples/flink-java-exa= mples-0.10-SNAPSHOT-WordCount.jar

You can easi= ly put this into your startup script, of course.

I= said almost possible because this requires a small fix in bin/flink. Aroun= d line 130 this line:
FLINK_CONF_DIR=3D$FLINK_ROOT_DIR_MANGLED/co= nf
needs to be replaced by this line:
if [ -z "= ;$FLINK_CONF_DIR" ]; then FLINK_CONF_DIR=3D$FLINK_ROOT_DIR_MANGLED/con= f; fi

(We will fix this in the upcoming version an= d the 0.9.1 bugfix release.)

Does this help? Let u= s know if you are not using the one-yarn-cluster-per-job mode, then we'= ll have to try to find another solution.

Forget th= e stuff about one-yarn-cluster-per-job but the other stuff should be right = what you want. I hope it helps.

Best,
Al= joscha

On Wed, 2= 6 Aug 2015 at 15:19 LINZ, Arnaud <ALINZ@bouyguestelecom.fr> wrote:

Ooops=E2=80=A6 Seems i= t was rather a write problem on the conf dir=E2=80=A6

Sorry, it works!

=C2=A0

BTW, it=E2=80=99s not = really nice to have an application write the configuration dir ; it=E2=80= =99s often a root protected directory in usr/lib/flink. Is there a parameter to put that file elsewhere ?=

=C2=A0

=C2=A0

De=C2=A0: Robert Metzger [mailto:rmetzger@apache.org]
Envoy=C3=A9=C2=A0: mercredi 26 ao=C3=BBt 2015 14:42
=C3=80=C2=A0: user@flink.apache.org
Objet=C2=A0: Re: [0.10-SNAPSHOT ] When naming yarn application (yarn= -session -nm), flink run without -m fails.

=C2=A0

Hi Arnaud,

=C2=A0

usually, you don't have to manually specify the = JobManager address manually with the -m argument, because it is reading it = from the conf/.yarn-session.properties file.

=C2=A0

Give me a few minutes to reproduce the issue.=

=C2=A0

On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud <ALINZ@bouyguest= elecom.fr> wrote:

Hi,
Using last nightly build, it seems that if you call yarn-session.sh with -nm option to give a nice application name, then you cannot submit a job with flink run without specify the ever changing -m <jobManager> address since it does not find it any longer.

Regards,

Arnaud

=C2=A0


L'int=C3=A9grit=C3=A9 de ce message n'=C3=A9tant pas assur=C3=A9e s= ur internet, la soci=C3=A9t=C3=A9 exp=C3=A9ditrice ne peut =C3=AAtre tenue = responsable de son contenu ni de ses pi=C3=A8ces jointes. Toute utilisation= ou diffusion non autoris=C3=A9e est interdite. Si vous n'=C3=AAtes pas= destinataire de ce message, merci de le d=C3=A9truire et d'avertir l'exp=C3=A9diteur.

The integrity of this message cannot be guaranteed on the Internet. The com= pany that sent this message cannot therefore be held liable for its content= nor attachments. Any unauthorized use or dissemination is prohibited. If y= ou are not the intended recipient of this message, then please delete it and notify the sender.

=C2=A0

--001a11c372eadd309f051e36cb8a--