flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jayant Ameta <wittyam...@gmail.com>
Subject Re: Performance of docker-flink
Date Thu, 07 Dec 2017 11:49:07 GMT
Thank you Gary.
I know that theoretically there shouldn't be any performance issue.
I was curious to know if any other users have tried out docker-flink and
whether they have faced/reported any performance hit. I would want real
time processing for some of the events, and was looking existing users'
experience with docker-flink.


Jayant Ameta

On Thu, Dec 7, 2017 at 4:37 PM, Gary Yao <gary@data-artisans.com> wrote:

> Hi Jayant,
>
> Running Flink in a Docker container should not have an impact on the
> performance
> in itself. Docker does not employ virtualization. To put it simply, Docker
> containers are processes on the host operating system that are isolated
> against
> each other using kernel features. See [1] for a more in-depth discussion.
>
> Whether the state of your Flink Application remains consistent when
> containers
> get restarted depends on many factors, such as whether you have
> checkpointing
> and JobManager HA enabled [2][3]. Also the checkpoint files still need to
> be
> available for job recovery after container restarts.
>
> If you want to use the docker images published under
> https://hub.docker.com/_/flink/, you probably want to overwrite the
> provided
> flink-conf.yaml by setting the FLINK_CONF_DIR environment variable to
> enable a
> fault tolerant setup.
>
> Best,
> Gary
>
> [1] https://stackoverflow.com/questions/21889053/what-is-
> the-runtime-performance-cost-of-a-docker-container
> [2] https://ci.apache.org/projects/flink/flink-docs-
> release-1.3/setup/checkpoints.html
> [3] https://ci.apache.org/projects/flink/flink-docs-
> release-1.3/setup/jobmanager_high_availability.html
>
> On Wed, Dec 6, 2017 at 9:43 AM, Jayant Ameta <wittyameta@gmail.com> wrote:
>
>> Hi,
>> I wanted to explore docker-flink (using Ceph for state backend). before
>> opting for a standalone cluster.
>>
>> Has there been any comparative studies on the performance of
>> docker-flink? Would the states be consistent and performant if the docker
>> containers go down and respawn frequently?
>>
>
>

Mime
View raw message