flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kostas Kloudas <k.klou...@data-artisans.com>
Subject Re: Deployment Architecture for Flink Applications
Date Wed, 22 Feb 2017 14:20:23 GMT
Hi CVP,

On how people use Flink, you can check this blogpost to see how Alibaba does it:
http://data-artisans.com/blink-flink-alibaba-search/ <http://data-artisans.com/blink-flink-alibaba-search/>

In addition, you can also find some more information on the matter on the talks from 
the last Flink Forwards conference: http://berlin.flink-forward.org/program/sessions/ <http://berlin.flink-forward.org/program/sessions/>

For example Netflix also shares some information here: 
http://berlin.flink-forward.org/kb_sessions/beaming-flink-to-the-cloud-netflix/ <http://berlin.flink-forward.org/kb_sessions/beaming-flink-to-the-cloud-netflix/>

Now for how things work under the hood, I will provide links to the Flink documentation. 
I hope that this will also help you figure out what fits your needs best:

For deployment and operations, the main resource is the Flink documentation, 
https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/cluster_setup.html <https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/cluster_setup.html>

and for what is about to come on that front, you can check out the FLIP-6 page:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077 <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077>

To dynamically scale your Flink job you have to take a savepoint and restart your job with
different parallelism.
You can find some details here https://www.slideshare.net/tillrohrmann/dynamic-scaling-how-apache-flink-adapts-to-changing-workloads
<https://www.slideshare.net/tillrohrmann/dynamic-scaling-how-apache-flink-adapts-to-changing-workloads>
, but unfortunately, this talk is a little bit outdated. We will update our documentation
on dynamic scaling soon.

For the Resource allocation and Job Scheduling, you can check the links I included for deployment
and operations,
and also this: https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/job_scheduling.html
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/job_scheduling.html>

For metrics and monitoring you can check here: https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html
<https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html>
and the related pages in the Debugging and monitoring section of the Flink documentation.

I hope this can help as a first step,
Kostas

> 
>     Right now our plan is to do a paper based study evaluating these options. 
>  
>     I'm sure lot of you guys in production/support would have encountered issues around
these. Can someone point out to blogs/research papers/material focussing on the approach taken
and the considerations for evaluation?
> 
>     Any help here is highly appreciated !
> 
> Best Regards
> CVP
>        

> On Feb 22, 2017, at 12:30 PM, Chakravarthy varaga <chakravarthyvp@gmail.com> wrote:
> 
> Hi Team,
> 
>     We are analysing different deployment options for managing Flink Jobs on AWS EC2
instances.
> 
>      Basically, the options (Resource Manangers) in front of us are using:
>      -> Standalone cluster
>      -> On YARN
>      -> Deploy using Mesos/Marthon
>      -> Deploy using Kubernetes/Docker
>      
>      The Resource Managers options are a bit confusing as we are unable to decide on
which one to go with. What we are looking at as inputs to our analysis is:
>     ->  Dynamic Scaling of resources
>     ->  Resource Allocation
>     ->  Jobs Scheduling 
>     ->  No-Downtime upgrades
>     ->  Monitoring & Metrics.
> 
>     Right now our plan is to do a paper based study evaluating these options. 
>  
>     I'm sure lot of you guys in production/support would have encountered issues around
these. Can someone point out to blogs/research papers/material focussing on the approach taken
and the considerations for evaluation?
> 
>     Any help here is highly appreciated !
> 
> Best Regards
> CVP
>        


Mime
View raw message