flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "LINZ, Arnaud" <AL...@bouyguestelecom.fr>
Subject RE: HA Mode and standalone containers compatibility ?
Date Thu, 03 Dec 2015 10:01:02 GMT
Yes, it does interfere, I do have additional task managers. My batch application comes in my
streaming cluster Flink’s GUI instead of creating its own container with its own GUI despite
the –m yarn-cluster option.

De : Till Rohrmann [mailto:trohrmann@apache.org]
Envoyé : jeudi 3 décembre 2015 10:36
À : user@flink.apache.org
Objet : Re: HA Mode and standalone containers compatibility ?

Hi Arnaud,

as long as you don't have HA activated for your batch jobs, HA shouldn't have an influence
on the batch execution. If it interferes, then you should see additional task manager connected
to the streaming cluster when you execute the batch job. Could you check that? Furthermore,
could you check that actually a second yarn application is started when you run the batch


On Thu, Dec 3, 2015 at 9:57 AM, LINZ, Arnaud <ALINZ@bouyguestelecom.fr<mailto:ALINZ@bouyguestelecom.fr>>


I have both streaming applications & batch applications. Since the memory needs are not
the same, I was using a long-living container for my streaming apps and new short-lived containers
for hosting each batch execution.

For that, I submit streaming jobs with "flink run"  and batch jobs with "flink run -m yarn-cluster"

This was working fine until I turned zookeeper HA mode on for my streaming applications.

Even if I don't set it up in the yaml flink configuration file, but with -D options on the
yarn_session.sh command line, now my batch jobs try to run in the streaming container, and
fails because of the lack of ressources.

My HA options are :

-Dyarn.application-attempts=10 -Drecovery.mode=zookeeper -Drecovery.zookeeper.quorum=h1r1en01:2181
-Drecovery.zookeeper.path.root=/flink  -Dstate.backend=filesystem -Dstate.backend.fs.checkpointdir=hdfs:///tmp/flink/checkpoints

Am I missing something ?

Best regards,



L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice
ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation
ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message,
merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent
this message cannot therefore be held liable for its content nor attachments. Any unauthorized
use or dissemination is prohibited. If you are not the intended recipient of this message,
then please delete it and notify the sender.

View raw message