flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Flink Kafka Streaming - Source [Bytes/records Received] and Sink [Bytes/records sent] show zero messages
Date Tue, 17 May 2016 07:29:41 GMT
Hi,
currently, Flink doesn't have code build in for throughput (rate) and
latency measurements. Its a planned feature for the future and some first
steps are done into that direction.
Check out this code for measuring throughput:
https://github.com/rmetzger/flink-state-watermarks/blob/master/src/main/java/com/dataartisans/utils/ThroughputLogger.java
(It will log the rates to the log file)

For the latency, I would attach timestamps to your records and measure the
delay by comparing the timestamps to the current system time (make sure the
clocks on all machines are in sync)


On Mon, May 16, 2016 at 11:43 PM, prateekarora <prateek.arora2k6@gmail.com>
wrote:

> thanks for the information .
> I want to collect some performance metrics like throughput , latency etc
> .thats why looking for these data to collect some information .
>
> how can i collect some performance data in flink  ?
>
> Regards
> Prateek
>
>
> On Mon, May 16, 2016 at 2:32 PM, Fabian Hueske-2 [via Apache Flink User
> Mailing List archive.] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=6945&i=0>> wrote:
>
>> Hi Prateek,
>>
>> the missing numbers are an artifact from how the stats are collected.
>> ATM, Flink does only collect these metrics for data which is sent over
>> connections *between* Flink operators.
>> Since sources and sinks connect to external systems (and not Flink
>> operators), the dash board does not show receiving byte for sources and
>> sent bytes for sinks.
>>
>> Best, Fabian
>>
>> 2016-05-16 23:02 GMT+02:00 prateekarora <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=6944&i=0>>:
>>
>>> Hi
>>>
>>> In my flink kafka streaming application i am fetching data from one topic
>>> and then process and sent to output topic . my application is working
>>> fine
>>> but  flink dashboard shows Source [Bytes/records Received]  and Sink
>>> [Bytes/records sent] is zero.
>>>
>>>
>>> Duration  Name                             Bytes            Records
>>>        Bytes
>>> Records       Status
>>>                                                   Received      Received
>>> sent           sent
>>> 3m 15s    Source: Custom Source   0 B              0
>>>  249 MB
>>> 3,540       RUNNING
>>>
>>>
>>> 3m 15s    Map -> Sink: Unnamed    203 MB         2,883         0 B
>>>             0
>>> RUNNING
>>>
>>>
>>>
>>> <
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n6941/flink-kafka.png
>>> >
>>>
>>> what is the reason for not showing correct number ?
>>>
>>> Regards
>>> Prateek
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Streaming-Source-Bytes-records-Received-and-Sink-Bytes-records-sent-show-zero-messages-tp6941.html
>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive at Nabble.com.
>>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Streaming-Source-Bytes-records-Received-and-Sink-Bytes-records-sent-show-zero-messages-tp6941p6944.html
>> To unsubscribe from Flink Kafka Streaming - Source [Bytes/records
>> Received] and Sink [Bytes/records sent] show zero messages, click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Flink Kafka Streaming - Source
> [Bytes/records Received] and Sink [Bytes/records sent] show zero messages
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Streaming-Source-Bytes-records-Received-and-Sink-Bytes-records-sent-show-zero-messages-tp6941p6945.html>
>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>

Mime
View raw message