flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Santos <dsan...@cryptolab.net>
Subject Re: Kafka Monitoring
Date Tue, 08 Nov 2016 12:16:38 GMT
Hi,

brokerPath is just optional.

Used if you want to have multile kafka clusters.

Each kafka cluster would connect to the same brokerPath.

Since I have multiple clusters I use the brokerPath.

 From the looks of it you dont. So never mind it doesn't matter.

You only have one zookeeper correct ?

And kafka and flink connects to that only zookeeper ?

Best Regards,

Daniel Santos


On 11/08/2016 11:18 AM, vinay patil wrote:
> Hi Daniel,
>
> Yes I have specified the zookeeper host in server.properties file , so 
> the broker is connected to zookeeper.
>
> https://kafka.apache.org/documentation#brokerconfigs 
> <https://kafka.apache.org/documentation#brokerconfigs> -> according to 
> this link, I guess all these configs are done in server.prop , so from 
> where did you get kafka09 as brokerPath ?
>
> this is my entry in server.prop file -> zookeeper.connect=localhost:2181
> Have you set this as zkhost:2181/kafka09 ?
>
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 4:27 PM, Daniel Santos [via Apache Flink User 
> Mailing List archive.] <[hidden email] 
> </user/SendEmail.jtp?type=node&node=9980&i=0>> wrote:
>
>     Hi,
>
>     Your kafka broker is connected to zookeeper I believe.
>
>     I am using kafka 0.9.0.1 my self too.
>
>     On kafka broker 0.9.0.1 I have configured the zookeeper connect to
>     a path, for instances :
>
>     zk1:2181,zk2:2181,zk3:2181/kafka09
>
>     https://kafka.apache.org/documentation#brokerconfigs
>     <https://kafka.apache.org/documentation#brokerconfigs>
>
>     Now on the flink side I would configure
>     "props.setProperty("zookeeper.connect", zkHosts)" the same
>     resulting in :
>
>
>     props.setProperty("zookeeperconnect",
>     "zk1:2181,zk2:2181,zk3:2181/kafka09")
>
>
>     That is what I mean by broker's path.
>
>     Best Regards,
>
>     Daniel Santos
>
>
>     On 11/08/2016 10:49 AM, vinay patil wrote:
>>     Hi Daniel,
>>
>>     I have the same properties set for the consumer and the same code
>>
>>     /brokerspath only needed if you have set it on kafka config/ -> I
>>     did not get this, do you mean to check the brokerspath in
>>     conf/server.properties file ? I have even tried by setting
>>     offset.storage property to zookeeeper, but still not getting the
>>     consumers listed
>>
>>     I am using Kafka 0.9.0.1
>>
>>
>>     Regards,
>>     Vinay Patil
>>
>>     On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink
>>     User Mailing List archive.] <[hidden email]
>>     <http:///user/SendEmail.jtp?type=node&node=9978&i=0>> wrote:
>>
>>         Hello,
>>
>>         This is my config.
>>
>>         On kafka props :
>>
>>             val props = new Properties()
>>
>>             props.setProperty("zookeeper.connect", zkHosts)
>>             props.setProperty("bootstrap.servers", kafHosts)
>>             props.setProperty("group.id <http://group.id>", "prod")
>>             props.setProperty("auto.offset.reset", "earliest")
>>
>>         Now for zkHosts beware that all your hosts quorum has to be
>>         included.
>>
>>         For instances you have zk1 and zk2 and zk3 to form a quorum.
>>
>>         Then it will result in zkHosts being ->
>>         zk1:2181,zk2:2181,zk3:2181/[brokerspath] .
>>
>>         brokerspath only needed if you have set it on kafka config.
>>         Ignore it otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" .
>>
>>         After that -> val source = env.addSource(new
>>         FlinkKafkaConsumer09[String](KAFKA_TOPIC, new
>>         SimpleStringSchema(), props))
>>
>>         Then on kafkamanager -> consumers I have the groupID prod.
>>
>>         Hope it helps.
>>
>>         Best Regards,
>>
>>         Daniel Santos
>>
>>         On 11/08/2016 08:45 AM, vinay patil wrote:
>>>         Hi Limbo,
>>>
>>>         I am using 0.9, I am not able to see updated results even
>>>         after refreshing.
>>>         There is some property that we have to set in order to make
>>>         this work
>>>
>>>         Regards,
>>>         Vinay Patil
>>>
>>>         On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink
>>>         User Mailing List archive.] <[hidden email]
>>>         <http:///user/SendEmail.jtp?type=node&node=9969&i=0>>
wrote:
>>>
>>>             I am using kafka 0.8, just refresh the page and you will
>>>             see the updated results, it’s not auto update.
>>>
>>>             There is our logstash consumer:
>>>
>>>
>>>
>>>             and this is our flink consumer:
>>>
>>>
>>>             I find the flink consumer just write the offset of the
>>>             kafka partition to zookeeper without owner and ids,
>>>             so we can’t find the consumer in the manager page, we
>>>             can only find the offset info.
>>>
>>>>             在 2016年11月8日,下午12:02,vinay patil <[hidden
email]
>>>>             <http:///user/SendEmail.jtp?type=node&node=9965&i=0>>
>>>>             写道:
>>>>
>>>>             Hi Limbo,
>>>>
>>>>             I can see the lag by using that URL, but the Lag there
>>>>             is not showing updated results, it does not change,
>>>>             also if you try to to change the consumer group value
>>>>             it will still show you the same value instead of saying
>>>>             consumer group does not exist or similar kind of error :)
>>>>
>>>>             According to documentation of 0.9.x the offsets are
>>>>             stored in Kafka, but we can set offset.storage property
>>>>             to zookeeper.
>>>>             Even by setting this I am not getting the consumer listed.
>>>>
>>>>             Kafka cli command also does not show this consumer
>>>>
>>>>             Regards,
>>>>             Vinay Patil
>>>>
>>>>             On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink
>>>>             User Mailing List archive.]<<a
>>>>             href="x-msg://4/user/SendEmail.jtp?type=node&amp;node=9964&amp;i=0"
>>>>             target="_top" rel="nofollow" link="external"
>>>>             class="">[hidden email]>wrote:
>>>>
>>>>                 Hi,
>>>>
>>>>                 I have the same problem, I think the reason is that
>>>>                 the consumer of flink use the low level API,
>>>>                 and when I type the group name in manager url I can
>>>>                 get the lag of the flink consumer, like this:
>>>>
>>>>                 http://your_manager_url/clusters/<kafka_name>/consumers/<consumer_name>/topic/<topic_name>/type/ZK
>>>>                 <http://your_manager_url/clusters/%3Ckafka_name%3E/consumers/%3Cconsumer_name%3E/topic/%3Ctopic_name%3E/type/ZK>
>>>>
>>>>>                 在 2016年11月8日,上午5:12,Daniel Santos <[hidden
>>>>>                 email]
>>>>>                 <http://user/SendEmail.jtp?type=node&node=9962&i=0>>
>>>>>                 写道:
>>>>>
>>>>>                 Hello,
>>>>>
>>>>>                 I have been using that setup.
>>>>>                 From my understanding, if one desires to see the
>>>>>                 offset being consumed by Flink on KafkaManger, one
>>>>>                 has to set it up with zookeeper. On 0.9 it will
>>>>>                 only serve as a view of progress.
>>>>>
>>>>>                 Basically what's mandatory on 0.8 is optional on
>>>>>                 0.9, and for viewing purposes only.
>>>>>
>>>>>                 Best Regards,
>>>>>                 Daniel Santos
>>>>>
>>>>>                 On November 7, 2016 7:13:54 PM GMT+00:00, Vinay
>>>>>                 Patil <[hidden email]
>>>>>                 <http://user/SendEmail.jtp?type=node&node=9962&i=1>>
>>>>>                 wrote:
>>>>>
>>>>>                     Hi,
>>>>>
>>>>>                     I am monitoring Kafka using KafkaManager for
>>>>>                     checking offset lag and other Kafka metrics,
>>>>>                     however I am not able to see the  consumers
>>>>>                     when I use FlinkKafkaConsumer , for
>>>>>                     console-consumer it shows them in the
>>>>>                     Consumers list.
>>>>>
>>>>>                     I have set the required parameters for the
>>>>>                     kafka consumer while running the application.
>>>>>
>>>>>                     Has anyone faced this issue ?
>>>>>                     I am using Kafka 0.9.0.1
>>>>>
>>>>>                     Regards,
>>>>>                     Vinay Patil
>>>>>
>>>>
>>>>
>>>>
>>>>                 ------------------------------------------------------------------------
>>>>                 If you reply to this email, your message will be
>>>>                 added to the discussion below:
>>>>                 http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html
>>>>                 <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html>
>>>>                 To start a new topic under Apache Flink User
>>>>                 Mailing List archive., email<a
>>>>                 href="x-msg://4/user/SendEmail.jtp?type=node&amp;node=9964&amp;i=1"
>>>>                 target="_top" rel="nofollow" link="external"
>>>>                 class="">[hidden email]
>>>>                 To unsubscribe from Apache Flink User Mailing List
>>>>                 archive.,click here.
>>>>                 NAML
>>>>                 <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>>
>>>>
>>>>             ------------------------------------------------------------------------
>>>>             View this message in context:Re: Kafka Monitoring
>>>>             <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
>>>>             Sent from theApache Flink User Mailing List archive.
>>>>             mailing list archive
>>>>             <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>atNabble.com
>>>>             <http://nabble.com/>.
>>>
>>>
>>>
>>>             ------------------------------------------------------------------------
>>>             If you reply to this email, your message will be added
>>>             to the discussion below:
>>>             http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html
>>>             <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html>
>>>
>>>             To start a new topic under Apache Flink User Mailing
>>>             List archive., email [hidden email]
>>>             <http:///user/SendEmail.jtp?type=node&node=9969&i=1>
>>>             To unsubscribe from Apache Flink User Mailing List
>>>             archive., click here.
>>>             NAML
>>>             <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>>
>>>         ------------------------------------------------------------------------
>>>         View this message in context: Re: Kafka Monitoring
>>>         <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
>>>         Sent from the Apache Flink User Mailing List archive.
>>>         mailing list archive
>>>         <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>>         at Nabble.com.
>>
>>
>>
>>         ------------------------------------------------------------------------
>>         If you reply to this email, your message will be added to the
>>         discussion below:
>>         http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9976.html
>>         <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9976.html>
>>
>>         To start a new topic under Apache Flink User Mailing List
>>         archive., email [hidden email]
>>         <http:///user/SendEmail.jtp?type=node&node=9978&i=1>
>>         To unsubscribe from Apache Flink User Mailing List archive.,
>>         click here.
>>         NAML
>>         <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>>
>>     ------------------------------------------------------------------------
>>     View this message in context: Re: Kafka Monitoring
>>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html>
>>     Sent from the Apache Flink User Mailing List archive. mailing
>>     list archive
>>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>     at Nabble.com.
>
>
>
>     ------------------------------------------------------------------------
>     If you reply to this email, your message will be added to the
>     discussion below:
>     http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9979.html
>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9979.html>
>
>     To start a new topic under Apache Flink User Mailing List
>     archive., email [hidden email]
>     </user/SendEmail.jtp?type=node&node=9980&i=1>
>     To unsubscribe from Apache Flink User Mailing List archive., click
>     here.
>     NAML
>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> ------------------------------------------------------------------------
> View this message in context: Re: Kafka Monitoring 
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9980.html>
> Sent from the Apache Flink User Mailing List archive. mailing list 
> archive 
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> 
> at Nabble.com.


Mime
View raw message