kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hu Xi <huxi...@hotmail.com>
Subject REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer
Date Tue, 21 Nov 2017 06:14:12 GMT
Hi Jun,


Seems the prefix that is used to be the unique Sensor name cannot be removed, so should we
keep the prefix?


________________________________
发件人: Jun Rao <jun@confluent.io>
发送时间: 2017年11月21日 3:55
收件人: dev@kafka.apache.org
主题: Re: 答复: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition
lead metrics to KafkaConsumer

Hi, Hu,

For the new partition level metrics that you are adding, it seems it's
better to just add the topic/partition tag instead of using them in the
prefix. For the existing lag metrics, we can fix them in KIP-225.

Thanks,

Jun

On Sun, Nov 19, 2017 at 10:31 PM, Hu Xi <huxi_2b@hotmail.com> wrote:

> Jun,
>
>
> Thanks for the comments. Do you think it'd better to add topic/partition
> tags for those metrics as well as keep the prefix? If those prefixes should
> really be removed, does this KIP need to do the same thing for `lag` ones?
>
> ________________________________
> 发件人: Jun Rao <jun@confluent.io>
> 发送时间: 2017年11月18日 8:55
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> per-partition lead metrics to KafkaConsumer
>
> Hi, Charly,
>
> Thanks for the input. It makes sense.
>
> Hi, Hu,
>
> Perhaps we can keep the per partition records-lead-min and records-lead-avg
> as you had before, but just add the topic and the partition as the tags
> instead of prefix of the metric name.
>
> Thanks,
>
> Jun
>
>
>
> On Wed, Nov 15, 2017 at 4:58 AM, charly molter <charly.molter@gmail.com>
> wrote:
>
> > Hi Jun, Jiangle,
> >
> > I'd just like to clarify that KIP-225 seems to be using per partition
> > metric the same way as KIP-223 seems to be doing.
> >
> > I believe avg and max are still necessary because the MetricsReporter
> > doesn't work in a "push" manner and the "Value" measurableStat will only
> > keep the last recorded entry.
> > Therefore a MetricsReporter usually polls to grab a current view with
> Value
> > this view is incomplete so it becomes not possible to compute the
> > Max/Min/Avg.
> > Max/Min/Avg uses SampledStats which work with a rolling window of samples
> > and therefore periodic polling would work.
> >
> > This is why I believe it's necessary to keep Avg, Min and Max for these
> > metrics as otherwise we wouldn't be able to recompute it in an external
> > monitoring system.
> >
> > Am I wrong thinking this?
> >
> > Thanks,
> > Charly
> >
> >
> > On Wed, Nov 15, 2017 at 2:02 AM, Jun Rao <jun@confluent.io> wrote:
> >
> > > Hi, Charly,
> > >
> > > Thanks for KIP-225. Your proposal looks reasonable.
> > >
> > > Hi, Jiangjie,
> > >
> > > Do you think the approach that KIP-225 proposes is better for exposing
> > the
> > > per partition metric? Also, do we really need the per partition
> > > record-lag-avg
> > > and record-lag-max? It seems that an external monitoring system can
> > always
> > > derive that from the per partition record-lag.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Nov 14, 2017 at 6:57 AM, charly molter <
> charly.molter@gmail.com>
> > > wrote:
> > >
> > > > Hi Jun, Hu,
> > > >
> > > > I have KIP-225 open for adding tags to records-lag:
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=74686649
> > > >
> > > > I have a patch more or less ready so I could probably get the fix
> > checked
> > > > in (after the vote) and you could build on top of it. Otherwise we
> > could
> > > > merge both KIPs if you want but they do sound different to me.
> > > >
> > > > Thanks!
> > > > Charly
> > > >
> > > > On Tue, Nov 14, 2017 at 11:42 AM, Hu Xi <huxi_2b@hotmail.com> wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > >
> > > > > Let me double confirm with your comments:
> > > > >
> > > > > 1 remove partition-level records-lead-avg and records-lead-min
> since
> > > they
> > > > > both can be deduced by external monitoring system.
> > > > >
> > > > > 2 Tag partition-level records-lead with topic&partition info
> > > > >
> > > > >
> > > > > If they are the case you expect, do we need to do the same thing
> for
> > > > those
> > > > > `lag` metrics? Seems partition-level records-lag metrics are not
> > tagged
> > > > > with topic&partition information  which might deserve a bug.
> > > > >
> > > > >
> > > > > huxihx
> > > > >
> > > > >
> > > > > ________________________________
> > > > > 发件人: Jun Rao <jun@confluent.io>
> > > > > 发送时间: 2017年11月14日 12:44
> > > > > 收件人: dev@kafka.apache.org
> > > > > 主题: Re: 答复: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition
> > > > > lead metrics to KafkaConsumer
> > > > >
> > > > > Hi, Hu,
> > > > >
> > > > > Currently, records-lag-max is an attribute for the mbean
> > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > id="{client-id}".
> > > > > So, it probably makes sense for records-lead-min to be an attribute
> > > under
> > > > > the same mbean.
> > > > >
> > > > > The partition level records-lead can probably be an attribute for
> the
> > > > mbean
> > > > > kafka.consumer:type=consumer-fetch-manager-metrics,client-
> > > > > id="{client-id}",topic=topic1,partition=0,
> > > > > where topic and partition are the tags. This matches the topic
> level
> > > > mbeans
> > > > > that we have in the consumer. I am not sure what the per partition
> > > level
> > > > > records-lead-min and records-lead-avg are. Are they the min/avg of
> > the
> > > > lead
> > > > > since the consumer is started? I am not sure we need those since
an
> > > > > external monitoring system can always derive them from
> records-lead.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 13, 2017 at 8:10 PM, Hu Xi <huxi_2b@hotmail.com>
> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > > Thanks for the feedback. Some things need to make sure.
> Currently,
> > > > these
> > > > > > new-added metrics follow the exact naming convention with those
> > 'lag'
> > > > > > counterparts, as shown below:
> > > > > >
> > > > > >
> > > > > > Consumer-level metric:
> > > > > >
> > > > > > records-lag-max ==> records-lead-min
> > > > > >
> > > > > >
> > > > > > Partition-level metrics:
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag          ==>
> > > <topic>-<partitionId>.
> > > > > > records-lead
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag-max ==> <topic>-<partitionId>.
> > > > > > records-lead-min
> > > > > >
> > > > > > <topic>-<partitionId>.records-lag-avg   ==>
> <topic>-<partitionId>.
> > > > > > records-lead-avg
> > > > > >
> > > > > >
> > > > > > Correct me if I am wrong, but what you mentioned
> `*records-lead-avg
> > > and
> > > > > > records-lead-min don't need the partition prefix since they
are
> > > > > aggregates
> > > > > > across all partitions*` seemed to break the naming rule above.
Do
> > we
> > > > > > still have to keep the same rule with the "lag" metrics?
> > > > > >
> > > > > >
> > > > > > huxihx
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ------------------------------
> > > > > > *发件人:* Jun Rao <jun@confluent.io>
> > > > > > *发送时间:* 2017年11月14日 1:48
> > > > > > *收件人:* dev@kafka.apache.org
> > > > > > *主题:* Re: [DISCUSS]KIP-223 - Add per-topic min lead and
> > per-partition
> > > > > > lead metrics to KafkaConsumer
> > > > > >
> > > > > > Hi, Hu,
> > > > > >
> > > > > > Thanks for the KIP. Looks good overall. Could you document the
> > mbean
> > > > name
> > > > > > for the new metrics? We probably want the name to be consistent
> > with
> > > > > > records-max-lag as described in
> > > > > > http://kafka.apache.org/documentation/#monitoring. Also, it
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>

Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®.
For an overview of a number of these areas in action, see this blog post.



> seems
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
[http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/documentation/#monitoring>

Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
kafka.apache.org
1.2 Use Cases. Here is a description of a few of the popular use cases for Apache Kafka®.
For an overview of a number of these areas in action, see this blog post.



> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > that
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > <http://kafka.apache.org/documentation/#monitoring>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > Apache Kafka <http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > [http://apache-kafka.org/images/apache-kafka.png
>
> [http://apache-kafka.org/images/apache-kafka.png]
>
> ]<http:
> > > > //kafka.apache.org/
> > > > > documentation/#monitoring>
> > > > >
> > > > > Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> [http://apache-kafka.org/images/apache-kafka.png]<http://kafka.apache.org/
> documentation/#monitoring>
>
> Apache Kafka<http://kafka.apache.org/documentation/#monitoring>
> kafka.apache.org
> 1.2 Use Cases. Here is a description of a few of the popular use cases for
> Apache Kafka®. For an overview of a number of these areas in action, see
> this blog post.
>
>
>
> > > > > kafka.apache.org
> > > > > 1.2 Use Cases. Here is a description of a few of the popular use
> > cases
> > > > for
> > > > > Apache Kafka®. For an overview of a number of these areas in
> action,
> > > see
> > > > > this blog post.
> > > > >
> > > > >
> > > > >
> > > > > > kafka.apache.org
> > > > > > 1.2 Use Cases. Here is a description of a few of the popular
use
> > > cases
> > > > > for
> > > > > > Apache Kafka®. For an overview of a number of these areas in
> > action,
> > > > see
> > > > > > this blog post.
> > > > > >
> > > > > >
> > > > > > records-lead-avg and records-lead-min don't need the partition
> > prefix
> > > > > since
> > > > > > they are aggregates across all partitions. For records-lead,
it
> > seems
> > > > > that
> > > > > > it's better to add the topic partition as a tag, instead of
as a
> > > prefix
> > > > > in
> > > > > > the metric name.
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Nov 9, 2017 at 1:03 AM, Hu Xi <huxi_2b@hotmail.com>
> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > >
> > > > > > > As per Jun Rao's suggestion, I opened up the KIP-223(
> > > > > > https://cwiki.apache.
> > > > > > > org/confluence/display/KAFKA/KIP-223+-+Add+per-topic+min+
> > > > > > > lead+and+per-partition+lead+metrics+to+KafkaConsumer)
> concerning
> > > > > adding
> > > > > > > new kinds of lag metrics for KafkaConsumer. Be free to
leave
> your
> > > > > > comments
> > > > > > > here. Thanks in advance.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Charly Molter
> > > >
> > >
> >
> >
> >
> > --
> > Charly Molter
> >
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message