Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
To: user@flink.apache.org
From: Ivan <email2liyang@gmail.com>
Subject: is there ways to enable checkpoint from flink-conf.yaml?
Message-ID: <64d344f7-e026-0645-a073-17ddfbf292cc@gmail.com>
Date: Wed, 26 Jul 2017 22:15:51 +0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0)
 Gecko/20100101 Thunderbird/45.8.0
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="------------7741AAE38357A56D2F9F14E3"
archived-at: Wed, 26 Jul 2017 14:16:02 -0000

This is a multi-part message in MIME format.
--------------7741AAE38357A56D2F9F14E3
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi , Flink users

we are using Flink as the runtime of our beam jobs which works great, 
recently we want to enable restart strategy 
<https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/restart_strategies.html> 
in our flink cluster, from the document I see restart strategy will only 
work when checkpointing is enabled. I'm trying to find out if it's 
possible to enable checkpointing from flink-conf.yaml which is 
equivalent to the call " 
flinkStreamEnv.enableCheckpointing(checkpointInterval);" in 
StreamExecutionEnvironment.

the reason we want to config it through flink-conf.yaml is that we use 
helm to create flink cluster ondemand for dedicated job which works 
great on kubernetes env. with beam if we want to enable checkpointing , 
we have to create FlinkPipelineOptions which is cross platform. (like  
you use Hibernate Session in JPA code). so we are trying to find a way 
to enable it from flink-conf.yaml .

sample flink-conf.yaml as below.

   flink-conf.yaml: |
     blob.server.port: 6124
     jobmanager.rpc.address: address-cache-flink-jobmanager
     jobmanager.rpc.port: 6123
     jobmanager.heap.mb: 256
     taskmanager.heap.mb: 756
     taskmanager.numberOfTaskSlots: 4
     parallelism.default: 16
     metrics.reporters: prom
     metrics.reporter.prom.class: 
org.apache.flink.metrics.prometheus.PrometheusReporter
     metrics.reporter.prom.port: 9100-9101
     metrics.reporter.prom.prefix: flink_jm_
     restart-strategy: fixed-delay
     restart-strategy.fixed-delay.attempts: 3
     restart-strategy.fixed-delay.delay: 300 s
     state.backend: filesystem
     state.backend.fs.checkpointdir: 
file:///var/nfs/ephem_store/flink/checkpoints


--------------7741AAE38357A56D2F9F14E3
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=utf-8">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <p>Hi , Flink users</p>
    <p>we are using Flink as the runtime of our beam jobs which works
      great, recently we want to enable <a
href="https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/restart_strategies.html">restart
        strategy</a> in our flink cluster, from the document I see
      restart strategy will only work when checkpointing is enabled. I'm
      trying to find out if it's possible to enable checkpointing from
      flink-conf.yaml which is equivalent to the call "
      <meta http-equiv="content-type" content="text/html; charset=utf-8">
      flinkStreamEnv.enableCheckpointing(checkpointInterval);" in
      StreamExecutionEnvironment.</p>
    the reason we want to config it through flink-conf.yaml is that we
    use helm to create flink cluster ondemand for dedicated job which
    works great on kubernetes env. with beam if we want to enable
    checkpointing , we have to create FlinkPipelineOptions which is
    cross platform. (like  you use Hibernate Session in JPA code). so we
    are trying to find a way to enable it from flink-conf.yaml .<br>
    <br>
    sample flink-conf.yaml as below.<br>
    <p>  flink-conf.yaml: |<br>
          blob.server.port: 6124<br>
          jobmanager.rpc.address: address-cache-flink-jobmanager<br>
          jobmanager.rpc.port: 6123<br>
          jobmanager.heap.mb: 256<br>
          taskmanager.heap.mb: 756<br>
          taskmanager.numberOfTaskSlots: 4<br>
          parallelism.default: 16<br>
          metrics.reporters: prom<br>
          metrics.reporter.prom.class:
      org.apache.flink.metrics.prometheus.PrometheusReporter<br>
          metrics.reporter.prom.port: 9100-9101<br>
          metrics.reporter.prom.prefix: flink_jm_<br>
          restart-strategy: fixed-delay<br>
          restart-strategy.fixed-delay.attempts: 3<br>
          restart-strategy.fixed-delay.delay: 300 s<br>
          state.backend: filesystem<br>
          state.backend.fs.checkpointdir:
      <a class="moz-txt-link-freetext" href="file:///var/nfs/ephem_store/flink/checkpoints">file:///var/nfs/ephem_store/flink/checkpoints</a><br>
      <meta http-equiv="content-type" content="text/html; charset=utf-8">
    </p>
  </body>
</html>

--------------7741AAE38357A56D2F9F14E3--