flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Web Dashboard reports 0 for sent/received bytes and records
Date Fri, 31 Mar 2017 18:51:28 GMT

there are collected and are accessible under the metrics tab.

Currently we can't display them directly since the backing job 
data-structure only sees tasks(aka chains) and
not the individual operators. There are some ongoing efforts to change 
that though, so we may be able
to improve the display in the web UI as well soon!


On 31.03.2017 16:46, Mohammad Kargar wrote:
> Hi Konstantin,
> What you are explaining here makes sense from a technical perspective! 
> However, functionality wise, I still think it makes sense for the 
> platform to collect and report, somehow, those metrics per job and 
> task out of the box.
> Thanks,
> Mohammad
> ---------------------------------------------------------------------------
> Hi Mohammad,
> that's normal. The source will never have a value other than zero in the
> "Records received" column, only in the "Records sent" column. But since
> your whole program is chained to one subtask, the source will never send
> anything to another subtask, because there is none.
> Same for the sink, it will only count values it receives from its
> preceding subtask, but due to chaining, there is no preceding subtask.
> Thus there will only be zeros in your dashboard.
> You can add your own Accumulators or Metrics and see the number of
> records in your operators through that, or you could also disable the
> chaining (but chaining is a good thing for performance, so you probably
> don't want to do that).
> Hope that helps,
> best regards
> Konstantin
> On 30.03.2017 16:48, Mohammad Kargar wrote:
> > Hi,
> >
> > I have a simple job where FlinkKafkaConsumer08 reads data from a topic
> > (with 8 partitions) and a custom Sink implementation inserts the data
> > into Accumulo database. I can verify that the data is getting inserted
> > into the db by adding simple system.out.println(...) to the invoke
> > method of the sink and also monitoring Accumulo stats. However Flink Web
> > Dashboard reports 0 for sent/received bytes and records (see attached
> > file for a snapshot). Any thoughts?
> >
> > *Hadoop platform*: Hortonworks Data Platform 2.5
> > *Flink*: flink-1.2.0-bin-hadoop27-scala_2.10
> >
> > Thanks,
> > Mohammad
> -- 
> Konstantin Gregor *konstantin.gregor@tngtech.com <mailto:konstantin.gregor@tngtech.com>
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082

View raw message