flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bowen Li <bowen...@offerupnow.com>
Subject How to register and send custom metrics from Flink?
Date Mon, 13 Mar 2017 07:48:55 GMT
Hi guys,
    I'm retrying to send some app related custom metrics from Flink to
Datadog via StatsD.

    I followed https://ci.apache.org/projects/flink/flink-docs-
release-1.2/monitoring/metrics.html to set up flink-conf.yaml and test code
like this

    // flink-conf.yaml

metrics.reporters: stsd
metrics.reporter.stsd.class: org.apache.flink.metrics.statsd.StatsDReporter
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125

metrics.scope.jm: bowen.jobmanager
metrics.scope.jm.job: bowen.jobmanager.bowen.job_name
metrics.scope.tm: bowen.taskmanager.<tm_id>

metrics.scope.tm.job: bowen.taskmanager.<tm_id>.<job_name>
metrics.scope.task: bowen.taskmanager.<tm_id>.<job_name>.<task_name>.<
metrics.scope.operator: bowen.taskmanager.<tm_id>.<

      // test code, by modifying WordCount example

public static final class Tokenizer extends
RichFlatMapFunction<String, Tuple2<String, Integer>> {
   private static final long serialVersionUID = 1L;

   private Counter counter;

   public void open(Configuration config) {
         .gauge("bowen.test.flink", new Gauge<Integer>() {
            public Integer getValue() {
               return 100;
         });  // test custom metrics

      counter = getRuntimeContext()
         .counter("bowen.flink.metric"); // test custom metrics

   public void flatMap(String value, Collector<Tuple2<String, Integer>> out)
         throws Exception {
      // normalize and split the line
      String[] tokens = value.toLowerCase().split("\\W+");

      // emit the pairs
      for (String token : tokens) {
         if (token.length() > 0) {
            out.collect(new Tuple2<String, Integer>(token, 1));


      I found my Datadog received all system scope metrics, but non of my
custom metric. I researched all night but gained no progress. What did I do
wrong? Flink is able to handle custom metrics right? I'd really appreciate
some guidance on sending custom metrics!

     Thank you very much!

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message