flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.
Date Wed, 26 Aug 2015 13:26:11 GMT
Hi Arnaud,
I think my answer to Gwenhaël could also be helpful to you:

are you using the one-yarn-cluster-per-job mode of Flink? I.e., you are
starting your Flink job with (from the doc):

flink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096
./examples/flink-java-examples-0.10-SNAPSHOT-WordCount.jar

If you are, then this is almost possible on the current version of Flink.
What you have to do is copy the conf directory of Flink to a separate
directory that is specific to your job. There you make your modifications
to the log configuration etc. Then, when you start your job you do this
instead:

export FLINK_CONF_DIR=/path/to/my/conf
flink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096
./examples/flink-java-examples-0.10-SNAPSHOT-WordCount.jar

You can easily put this into your startup script, of course.

I said almost possible because this requires a small fix in bin/flink.
Around line 130 this line:
FLINK_CONF_DIR=$FLINK_ROOT_DIR_MANGLED/conf
needs to be replaced by this line:
if [ -z "$FLINK_CONF_DIR" ]; then
FLINK_CONF_DIR=$FLINK_ROOT_DIR_MANGLED/conf; fi

(We will fix this in the upcoming version and the 0.9.1 bugfix release.)

Does this help? Let us know if you are not using the
one-yarn-cluster-per-job mode, then we'll have to try to find another
solution.

Forget the stuff about one-yarn-cluster-per-job but the other stuff should
be right what you want. I hope it helps.

Best,
Aljoscha

On Wed, 26 Aug 2015 at 15:19 LINZ, Arnaud <ALINZ@bouyguestelecom.fr> wrote:

> Ooops… Seems it was rather a write problem on the conf dir…
>
> Sorry, it works!
>
>
>
> BTW, it’s not really nice to have an application write the configuration
> dir ; it’s often a root protected directory in usr/lib/flink. Is there a
> parameter to put that file elsewhere ?
>
>
>
>
>
> *De :* Robert Metzger [mailto:rmetzger@apache.org]
> *Envoyé :* mercredi 26 août 2015 14:42
> *À :* user@flink.apache.org
> *Objet :* Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session
> -nm), flink run without -m fails.
>
>
>
> Hi Arnaud,
>
>
>
> usually, you don't have to manually specify the JobManager address
> manually with the -m argument, because it is reading it from the
> conf/.yarn-session.properties file.
>
>
>
> Give me a few minutes to reproduce the issue.
>
>
>
> On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud <ALINZ@bouyguestelecom.fr>
> wrote:
>
> Hi,
> Using last nightly build, it seems that if you call yarn-session.sh with
> -nm option to give a nice application name, then you cannot submit a job
> with flink run without specify the ever changing -m <jobManager> address
> since it does not find it any longer.
>
> Regards,
>
> Arnaud
>
>
> ------------------------------
>
>
> L'intégrité de ce message n'étant pas assurée sur internet, la société
> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si
> vous n'êtes pas destinataire de ce message, merci de le détruire et
> d'avertir l'expéditeur.
>
> The integrity of this message cannot be guaranteed on the Internet. The
> company that sent this message cannot therefore be held liable for its
> content nor attachments. Any unauthorized use or dissemination is
> prohibited. If you are not the intended recipient of this message, then
> please delete it and notify the sender.
>
>
>

Mime
View raw message