flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eron Wright <eronwri...@gmail.com>
Subject Re: Automate Job submission with container
Date Tue, 15 Aug 2017 20:33:55 GMT
Flink 1.3 relies on a two-step approach of deploying the cluster on Mesos
and then deploying the job.   The cron task seems like a good approach;
maybe retry until the job is successfully deployed (using the 'detached'
option assumedly).   One complication is that the slots take some time to
come online, and the JM will reject the job submission until sufficient
slots are available.

One of the goals of the FLIP-6 improvement project is to allow for a
one-step deployment of cluster plus job. In the meantime please use a
workaround like you described.

Thanks

On Mon, Aug 14, 2017 at 6:04 PM, Biswajit Das <biswajit.ds@gmail.com> wrote:

> Hi There,
>
> A few weeks back I have posted here regarding flink docker container
> running on mesos. I'm able to run the same successfully; I'm heading
> towards full CI/CD deployment with the marathon, most of our deployment is
> automated via the marathon, and sometimes even I trigger metrics based auto
> scale from the marathon or another scaling service or move service with AWS
> spot instance randomly.
> Now how do I trigger the job automatically inside the container once JM &
> TM is up . I have tested few option and just wanted to validate with the
> group here. I have uber app jar already bundled inside the Docker container
> .
>
> Option .
>    1. Schedule a one-time cron like task inside the container with flink
> run -d and some delay.
> 2.  Submit via REST, but it complicates the process as I have to find app
> DNS from my CI although I have this setup with Bamboo and HAProxy just want
> to avoid this complicated process.( given I have many apps running )
>
> I'm sure someone already came across the similar problem, please let me
> know if any one has any inputs or suggestion.
>
> ~ Biswajit
>

Mime
View raw message