flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Santos <dsan...@cryptolab.net>
Subject Re: Kafka Monitoring
Date Tue, 08 Nov 2016 11:07:41 GMT
Hi,

Your kafka broker is connected to zookeeper I believe.

I am using kafka 0.9.0.1 my self too.

On kafka broker 0.9.0.1 I have configured the zookeeper connect to a 
path, for instances :

zk1:2181,zk2:2181,zk3:2181/kafka09

https://kafka.apache.org/documentation#brokerconfigs

Now on the flink side I would configure 
"props.setProperty("zookeeper.connect", zkHosts)" the same resulting in :


props.setProperty("zookeeperconnect", "zk1:2181,zk2:2181,zk3:2181/kafka09")


That is what I mean by broker's path.

Best Regards,

Daniel Santos


On 11/08/2016 10:49 AM, vinay patil wrote:
> Hi Daniel,
>
> I have the same properties set for the consumer and the same code
>
> /brokerspath only needed if you have set it on kafka config/ -> I did 
> not get this, do you mean to check the brokerspath in 
> conf/server.properties file ? I have even tried by setting 
> offset.storage property to zookeeeper, but still not getting the 
> consumers listed
>
> I am using Kafka 0.9.0.1
>
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink User 
> Mailing List archive.] <[hidden email] 
> </user/SendEmail.jtp?type=node&node=9978&i=0>> wrote:
>
>     Hello,
>
>     This is my config.
>
>     On kafka props :
>
>         val props = new Properties()
>
>         props.setProperty("zookeeper.connect", zkHosts)
>         props.setProperty("bootstrap.servers", kafHosts)
>         props.setProperty("group.id <http://group.id>", "prod")
>         props.setProperty("auto.offset.reset", "earliest")
>
>     Now for zkHosts beware that all your hosts quorum has to be included.
>
>     For instances you have zk1 and zk2 and zk3 to form a quorum.
>
>     Then it will result in zkHosts being ->
>     zk1:2181,zk2:2181,zk3:2181/[brokerspath] .
>
>     brokerspath only needed if you have set it on kafka config. Ignore
>     it otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" .
>
>     After that -> val source = env.addSource(new
>     FlinkKafkaConsumer09[String](KAFKA_TOPIC, new
>     SimpleStringSchema(), props))
>
>     Then on kafkamanager -> consumers I have the groupID prod.
>
>     Hope it helps.
>
>     Best Regards,
>
>     Daniel Santos
>
>     On 11/08/2016 08:45 AM, vinay patil wrote:
>>     Hi Limbo,
>>
>>     I am using 0.9, I am not able to see updated results even after
>>     refreshing.
>>     There is some property that we have to set in order to make this work
>>
>>     Regards,
>>     Vinay Patil
>>
>>     On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User
>>     Mailing List archive.] <[hidden email]
>>     <http:///user/SendEmail.jtp?type=node&node=9969&i=0>> wrote:
>>
>>         I am using kafka 0.8, just refresh the page and you will see
>>         the updated results, it’s not auto update.
>>
>>         There is our logstash consumer:
>>
>>
>>
>>         and this is our flink consumer:
>>
>>
>>         I find the flink consumer just write the offset of the kafka
>>         partition to zookeeper without owner and ids,
>>         so we can’t find the consumer in the manager page, we can
>>         only find the offset info.
>>
>>>         在 2016年11月8日,下午12:02,vinay patil <[hidden email]
>>>         <http:///user/SendEmail.jtp?type=node&node=9965&i=0>>
写道:
>>>
>>>         Hi Limbo,
>>>
>>>         I can see the lag by using that URL, but the Lag there is
>>>         not showing updated results, it does not change, also if you
>>>         try to to change the consumer group value it will still show
>>>         you the same value instead of saying consumer group does not
>>>         exist or similar kind of error :)
>>>
>>>         According to documentation of 0.9.x the offsets are stored
>>>         in Kafka, but we can set offset.storage property to zookeeper.
>>>         Even by setting this I am not getting the consumer listed.
>>>
>>>         Kafka cli command also does not show this consumer
>>>
>>>         Regards,
>>>         Vinay Patil
>>>
>>>         On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User
>>>         Mailing List archive.]<<a
>>>         href="x-msg://4/user/SendEmail.jtp?type=node&amp;node=9964&amp;i=0"
>>>         target="_top" rel="nofollow" link="external"
>>>         class="">[hidden email]>wrote:
>>>
>>>             Hi,
>>>
>>>             I have the same problem, I think the reason is that the
>>>             consumer of flink use the low level API,
>>>             and when I type the group name in manager url I can get
>>>             the lag of the flink consumer, like this:
>>>
>>>             http://your_manager_url/clusters/<kafka_name>/consumers/<consumer_name>/topic/<topic_name>/type/ZK
>>>             <http://your_manager_url/clusters/%3Ckafka_name%3E/consumers/%3Cconsumer_name%3E/topic/%3Ctopic_name%3E/type/ZK>
>>>
>>>>             在 2016年11月8日,上午5:12,Daniel Santos <[hidden
email]
>>>>             <http://user/SendEmail.jtp?type=node&node=9962&i=0>>
写道:
>>>>
>>>>             Hello,
>>>>
>>>>             I have been using that setup.
>>>>             From my understanding, if one desires to see the offset
>>>>             being consumed by Flink on KafkaManger, one has to set
>>>>             it up with zookeeper. On 0.9 it will only serve as a
>>>>             view of progress.
>>>>
>>>>             Basically what's mandatory on 0.8 is optional on 0.9,
>>>>             and for viewing purposes only.
>>>>
>>>>             Best Regards,
>>>>             Daniel Santos
>>>>
>>>>             On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil
>>>>             <[hidden email]
>>>>             <http://user/SendEmail.jtp?type=node&node=9962&i=1>>
>>>>             wrote:
>>>>
>>>>                 Hi,
>>>>
>>>>                 I am monitoring Kafka using KafkaManager for
>>>>                 checking offset lag and other Kafka metrics,
>>>>                 however I am not able to see the  consumers when I
>>>>                 use FlinkKafkaConsumer , for console-consumer it
>>>>                 shows them in the Consumers list.
>>>>
>>>>                 I have set the required parameters for the kafka
>>>>                 consumer while running the application.
>>>>
>>>>                 Has anyone faced this issue ?
>>>>                 I am using Kafka 0.9.0.1
>>>>
>>>>                 Regards,
>>>>                 Vinay Patil
>>>>
>>>
>>>
>>>
>>>             ------------------------------------------------------------------------
>>>             If you reply to this email, your message will be added
>>>             to the discussion below:
>>>             http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html
>>>             <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html>
>>>             To start a new topic under Apache Flink User Mailing
>>>             List archive., email<a
>>>             href="x-msg://4/user/SendEmail.jtp?type=node&amp;node=9964&amp;i=1"
>>>             target="_top" rel="nofollow" link="external"
>>>             class="">[hidden email]
>>>             To unsubscribe from Apache Flink User Mailing List
>>>             archive.,click here.
>>>             NAML
>>>             <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>>         ------------------------------------------------------------------------
>>>         View this message in context:Re: Kafka Monitoring
>>>         <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
>>>         Sent from theApache Flink User Mailing List archive. mailing
>>>         list archive
>>>         <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>atNabble.com
>>>         <http://nabble.com/>.
>>
>>
>>
>>         ------------------------------------------------------------------------
>>         If you reply to this email, your message will be added to the
>>         discussion below:
>>         http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html
>>         <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html>
>>
>>         To start a new topic under Apache Flink User Mailing List
>>         archive., email [hidden email]
>>         <http:///user/SendEmail.jtp?type=node&node=9969&i=1>
>>         To unsubscribe from Apache Flink User Mailing List archive.,
>>         click here.
>>         NAML
>>         <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>>
>>     ------------------------------------------------------------------------
>>     View this message in context: Re: Kafka Monitoring
>>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
>>     Sent from the Apache Flink User Mailing List archive. mailing
>>     list archive
>>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>     at Nabble.com.
>
>
>
>     ------------------------------------------------------------------------
>     If you reply to this email, your message will be added to the
>     discussion below:
>     http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9976.html
>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9976.html>
>
>     To start a new topic under Apache Flink User Mailing List
>     archive., email [hidden email]
>     </user/SendEmail.jtp?type=node&node=9978&i=1>
>     To unsubscribe from Apache Flink User Mailing List archive., click
>     here.
>     NAML
>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> ------------------------------------------------------------------------
> View this message in context: Re: Kafka Monitoring 
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html>
> Sent from the Apache Flink User Mailing List archive. mailing list 
> archive 
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> 
> at Nabble.com.


Mime
View raw message