flink-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ar...@apache.org
Subject [flink] 02/03: [FLINK-23817][docs] Add metric description to Kafka source.
Date Wed, 01 Sep 2021 07:14:05 GMT
This is an automated email from the ASF dual-hosted git repository.

arvid pushed a commit to branch release-1.14
in repository https://gitbox.apache.org/repos/asf/flink.git

commit 4062e1b24e8becf398fe76bd40a407421c21e1de
Author: Arvid Heise <arvid@ververica.com>
AuthorDate: Tue Aug 31 08:58:56 2021 +0200

    [FLINK-23817][docs] Add metric description to Kafka source.
---
 docs/content/docs/connectors/datastream/kafka.md | 86 +++++++++++++++++++-----
 docs/content/docs/ops/metrics.md                 | 47 +------------
 2 files changed, 72 insertions(+), 61 deletions(-)

diff --git a/docs/content/docs/connectors/datastream/kafka.md b/docs/content/docs/connectors/datastream/kafka.md
index 8f2c56e..d49303a 100644
--- a/docs/content/docs/connectors/datastream/kafka.md
+++ b/docs/content/docs/connectors/datastream/kafka.md
@@ -212,26 +212,82 @@ Note that Kafka source does **NOT** rely on committed offsets for fault
toleranc
 is only for exposing the progress of consumer and consuming group for monitoring.
 
 ### Monitoring
-Kafka source exposes metrics in Flink's metric group for monitoring and diagnosing.
+
+Kafka source exposes the following metrics in the respective [scope]({{< ref "docs/ops/metrics"
>}}/#scope).
+
 #### Scope of Metric
-All metrics of Kafka source reader are registered under group ```KafkaSourceReader```, which
is a 
-child group of operator metric group. Metrics related to a specific topic partition will
be registered
-in the group ```KafkaSourceReader.topic.<topic_name>.partition.<partition_id>```.
 
-For example, current consuming offset of topic "my-topic" and partition 1 will be reported
in metric: 
-```<some_parent_groups>.operator.KafkaSourceReader.topic.my-topic.partition.1.currentOffset```
,
+<table class="table table-bordered">
+  <thead>
+    <tr>
+      <th class="text-left" style="width: 15%">Scope</th>
+      <th class="text-left" style="width: 18%">Metrics</th>
+      <th class="text-left" style="width: 18%">User Variables</th>
+      <th class="text-left" style="width: 39%">Description</th>
+      <th class="text-left" style="width: 10%">Type</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+        <th rowspan="8">Operator</th>
+        <td>currentEmitEventTimeLag</td>
+        <td>n/a</td>
+        <td>The time span from the record event timestamp to the time the record is
emitted by the source connector┬╣: <code>currentEmitEventTimeLag = EmitTime - EventTime.</code></td>
+        <td>Gauge</td>
+    </tr>
+    <tr>
+        <td>watermarkLag</td>
+        <td>n/a</td>
+        <td>The time span that the watermark lags behind the wall clock time: <code>watermarkLag
= CurrentTime - Watermark</code></td>
+        <td>Gauge</td>
+    </tr>
+    <tr>
+        <td>sourceIdleTime</td>
+        <td>n/a</td>
+        <td>The time span that the source has not processed any record: <code>sourceIdleTime
= CurrentTime - LastRecordProcessTime</code></td>
+        <td>Gauge</td>
+    </tr>
+    <tr>
+        <td>pendingRecords</td>
+        <td>n/a</td>
+        <td>The number of records that have not been fetched by the source. e.g. the
available records after the consumer offset in a Kafka partition.</td>
+        <td>Gauge</td>
+    </tr>
+    <tr>
+      <td>KafkaSourceReader.commitsSucceeded</td>
+      <td>n/a</td>
+      <td>The total number of successful offset commits to Kafka, if offset committing
is turned on and checkpointing is enabled.</td>
+      <td>Counter</td>
+    </tr>
+    <tr>
+       <td>KafkaSourceReader.commitsFailed</td>
+       <td>n/a</td>
+       <td>The total number of offset commit failures to Kafka, if offset committing
is
+       turned on and checkpointing is enabled. Note that committing offsets back to Kafka
+       is only a means to expose consumer progress, so a commit failure does not affect
+       the integrity of Flink's checkpointed partition offsets.</td>
+       <td>Counter</td>
+    </tr>
+    <tr>
+       <td>KafkaSourceReader.committedOffsets</td>
+       <td>topic, partition</td>
+       <td>The last successfully committed offsets to Kafka, for each partition.
+       A particular partition's metric can be specified by topic name and partition id.</td>
+       <td>Gauge</td>
+    </tr>
+    <tr>
+      <td>KafkaSourceReader.currentOffsets</td>
+      <td>topic, partition</td>
+      <td>The consumer's current read offset, for each partition. A particular
+      partition's metric can be specified by topic name and partition id.</td>
+      <td>Gauge</td>
+    </tr>
+  </tbody>
+</table>
 
-and number of successful commits will be reported in metric:
-```<some_parent_groups>.operator.KafkaSourceReader.commitsSucceeded``` .
 
-#### List of Metrics
+┬╣ This metric is an instantaneous value recorded for the last processed record. This metric
is provided because latency histogram could be expensive. The instantaneous latency value
is usually a good enough indication of the latency. 
 
-|    Metric Name   |                   Description                   |       Scope      
|
-|:----------------:|:-----------------------------------------------:|:-----------------:|
-|   currentOffset  | Current consuming offset of the topic partition |   TopicPartition 
|
-|  committedOffset | Committed offset of the topic partition         |   TopicPartition 
|
-| commitsSucceeded | Number of successful commits                    | KafkaSourceReader
|
-|   commitsFailed  | Number of failed commits                        | KafkaSourceReader
|
 
 #### Kafka Consumer Metrics
 All metrics of Kafka consumer are also registered under group ```KafkaSourceReader.KafkaConsumer```.
diff --git a/docs/content/docs/ops/metrics.md b/docs/content/docs/ops/metrics.md
index 53b59d9..1580136 100644
--- a/docs/content/docs/ops/metrics.md
+++ b/docs/content/docs/ops/metrics.md
@@ -1345,52 +1345,7 @@ Certain RocksDB native metrics are available but disabled by default,
you can fi
 ### Connectors
 
 #### Kafka Connectors
-<table class="table table-bordered">
-  <thead>
-    <tr>
-      <th class="text-left" style="width: 15%">Scope</th>
-      <th class="text-left" style="width: 18%">Metrics</th>
-      <th class="text-left" style="width: 18%">User Variables</th>
-      <th class="text-left" style="width: 39%">Description</th>
-      <th class="text-left" style="width: 10%">Type</th>
-    </tr>
-  </thead>
-  <tbody>
-    <tr>
-      <th rowspan="1">Operator</th>
-      <td>commitsSucceeded</td>
-      <td>n/a</td>
-      <td>The total number of successful offset commits to Kafka, if offset committing
is turned on and checkpointing is enabled.</td>
-      <td>Counter</td>
-    </tr>
-    <tr>
-       <th rowspan="1">Operator</th>
-       <td>commitsFailed</td>
-       <td>n/a</td>
-       <td>The total number of offset commit failures to Kafka, if offset committing
is
-       turned on and checkpointing is enabled. Note that committing offsets back to Kafka
-       is only a means to expose consumer progress, so a commit failure does not affect
-       the integrity of Flink's checkpointed partition offsets.</td>
-       <td>Counter</td>
-    </tr>
-    <tr>
-       <th rowspan="1">Operator</th>
-       <td>committedOffsets</td>
-       <td>topic, partition</td>
-       <td>The last successfully committed offsets to Kafka, for each partition.
-       A particular partition's metric can be specified by topic name and partition id.</td>
-       <td>Gauge</td>
-    </tr>
-    <tr>
-      <th rowspan="1">Operator</th>
-      <td>currentOffsets</td>
-      <td>topic, partition</td>
-      <td>The consumer's current read offset, for each partition. A particular
-      partition's metric can be specified by topic name and partition id.</td>
-      <td>Gauge</td>
-    </tr>
-  </tbody>
-</table>
+Please refer to [Kafka monitoring]({{< ref "docs/connectors/datastream/kafka" >}}/#monitoring).
 
 #### Kinesis Connectors
 <table class="table table-bordered">

Mime
View raw message