flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From limbo <468137...@qq.com>
Subject Re: Kafka Monitoring
Date Tue, 08 Nov 2016 07:11:19 GMT
I am using kafka 0.8, just refresh the page and you will see the updated results, it’s not
auto update.

There is our logstash consumer:



and this is our flink consumer:



I find the flink consumer just write the offset of the kafka partition to zookeeper without
owner and ids, 
so we can’t find the consumer in the manager page, we can only find the offset info.

> 在 2016年11月8日,下午12:02,vinay patil <vinay18.patil@gmail.com> 写道:
> 
> Hi Limbo,
> 
> I can see the lag by using that URL, but the Lag there is not showing updated results,
it does not change, also if you try to to change the consumer group value it will still show
you the same value instead of saying consumer group does not exist or similar kind of error
:)
> 
> According to documentation of 0.9.x the offsets are stored in Kafka, but we can set offset.storage
property to zookeeper.
> Even by setting this I am not getting the consumer listed.
> 
> Kafka cli command also does not show this consumer 
> 
> Regards,
> Vinay Patil
> 
> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User Mailing List archive.] <[hidden
email] <x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=0>> wrote:
> Hi,
> 
> I have the same problem, I think the reason is that the consumer of flink use the low
level API,
> and when I type the group name in manager url I can get the lag of the flink consumer,
like this: 
> 
> http://your_manager_url/clusters/<kafka_name>/consumers/<consumer_name>/topic/<topic_name>/type/ZK
<http://your_manager_url/clusters/%3Ckafka_name%3E/consumers/%3Cconsumer_name%3E/topic/%3Ctopic_name%3E/type/ZK>
> 
>> 在 2016年11月8日,上午5:12,Daniel Santos <[hidden email] <http://user/SendEmail.jtp?type=node&node=9962&i=0>>
写道:
>> 
>> Hello, 
>> 
>> I have been using that setup. 
>> From my understanding, if one desires to see the offset being consumed by Flink on
KafkaManger, one has to set it up with zookeeper. On 0.9 it will only serve as a view of progress.

>> 
>> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing purposes only.

>> 
>> Best Regards, 
>> Daniel Santos 
>> 
>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden email] <http://user/SendEmail.jtp?type=node&node=9962&i=1>>
wrote:
>> Hi,
>> 
>> I am monitoring Kafka using KafkaManager for checking offset lag and other Kafka
metrics, however I am not able to see the  consumers when I use FlinkKafkaConsumer , for console-consumer
it shows them in the Consumers list.
>> 
>> I have set the required parameters for the kafka consumer while running the application.
>> 
>> Has anyone faced this issue ? 
>> I am using Kafka 0.9.0.1
>> 
>> Regards,
>> Vinay Patil
> 
> 
> 
> If you reply to this email, your message will be added to the discussion below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html>
> To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
<x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1> 
> To unsubscribe from Apache Flink User Mailing List archive., click here <applewebdata://911FFAC4-9E12-48DB-AC3D-1BDD5A374626>.
> NAML <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> 
> View this message in context: Re: Kafka Monitoring <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
at Nabble.com <http://nabble.com/>.


Mime
View raw message