flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Santos <dsan...@cryptolab.net>
Subject Re: Kafka Monitoring
Date Tue, 08 Nov 2016 19:11:58 GMT
Hello,

On flink do you have the checkpoint enabled ?

env.enableCheckpointing(interval = CHKPOINT_INTERVAL)

Regards,

Daniel Santos


On 11/08/2016 12:30 PM, vinay patil wrote:
> Yes Kafka and Flink connect to that zookeeper only.
>
> Not sure why it is not listing the consumer
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 5:36 PM, Daniel Santos [via Apache Flink User 
> Mailing List archive.] <[hidden email] 
> </user/SendEmail.jtp?type=node&node=9983&i=0>> wrote:
>
>     Hi,
>
>     brokerPath is just optional.
>
>     Used if you want to have multile kafka clusters.
>
>     Each kafka cluster would connect to the same brokerPath.
>
>     Since I have multiple clusters I use the brokerPath.
>
>     From the looks of it you dont. So never mind it doesn't matter.
>
>     You only have one zookeeper correct ?
>
>     And kafka and flink connects to that only zookeeper ?
>
>     Best Regards,
>
>     Daniel Santos
>
>
>     On 11/08/2016 11:18 AM, vinay patil wrote:
>>     Hi Daniel,
>>
>>     Yes I have specified the zookeeper host in server.properties file
>>     , so the broker is connected to zookeeper.
>>
>>     https://kafka.apache.org/documentation#brokerconfigs
>>     <https://kafka.apache.org/documentation#brokerconfigs> ->
>>     according to this link, I guess all these configs are done in
>>     server.prop , so from where did you get kafka09 as brokerPath ?
>>
>>     this is my entry in server.prop file ->
>>     zookeeper.connect=localhost:2181
>>     Have you set this as zkhost:2181/kafka09 ?
>>
>>
>>     Regards,
>>     Vinay Patil
>>
>>     On Tue, Nov 8, 2016 at 4:27 PM, Daniel Santos [via Apache Flink
>>     User Mailing List archive.] <[hidden email]
>>     <http:///user/SendEmail.jtp?type=node&node=9980&i=0>> wrote:
>>
>>         Hi,
>>
>>         Your kafka broker is connected to zookeeper I believe.
>>
>>         I am using kafka 0.9.0.1 my self too.
>>
>>         On kafka broker 0.9.0.1 I have configured the zookeeper
>>         connect to a path, for instances :
>>
>>         zk1:2181,zk2:2181,zk3:2181/kafka09
>>
>>         https://kafka.apache.org/documentation#brokerconfigs
>>         <https://kafka.apache.org/documentation#brokerconfigs>
>>
>>         Now on the flink side I would configure
>>         "props.setProperty("zookeeper.connect", zkHosts)" the same
>>         resulting in :
>>
>>
>>         props.setProperty("zookeeperconnect",
>>         "zk1:2181,zk2:2181,zk3:2181/kafka09")
>>
>>
>>         That is what I mean by broker's path.
>>
>>         Best Regards,
>>
>>         Daniel Santos
>>
>>
>>         On 11/08/2016 10:49 AM, vinay patil wrote:
>>>         Hi Daniel,
>>>
>>>         I have the same properties set for the consumer and the same
>>>         code
>>>
>>>         /brokerspath only needed if you have set it on kafka config/
>>>         -> I did not get this, do you mean to check the brokerspath
>>>         in conf/server.properties file ? I have even tried by
>>>         setting offset.storage property to zookeeeper, but still not
>>>         getting the consumers listed
>>>
>>>         I am using Kafka 0.9.0.1
>>>
>>>
>>>         Regards,
>>>         Vinay Patil
>>>
>>>         On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache
>>>         Flink User Mailing List archive.] <[hidden email]
>>>         <http:///user/SendEmail.jtp?type=node&node=9978&i=0>>
wrote:
>>>
>>>             Hello,
>>>
>>>             This is my config.
>>>
>>>             On kafka props :
>>>
>>>                 val props = new Properties()
>>>
>>>                 props.setProperty("zookeeper.connect", zkHosts)
>>>                 props.setProperty("bootstrap.servers", kafHosts)
>>>                 props.setProperty("group.id <http://group.id>", "prod")
>>>                 props.setProperty("auto.offset.reset", "earliest")
>>>
>>>             Now for zkHosts beware that all your hosts quorum has to
>>>             be included.
>>>
>>>             For instances you have zk1 and zk2 and zk3 to form a quorum.
>>>
>>>             Then it will result in zkHosts being ->
>>>             zk1:2181,zk2:2181,zk3:2181/[brokerspath] .
>>>
>>>             brokerspath only needed if you have set it on kafka
>>>             config. Ignore it otherwise, resulting in
>>>             "zk1:2181,zk2:2181,zk3:2181" .
>>>
>>>             After that -> val source = env.addSource(new
>>>             FlinkKafkaConsumer09[String](KAFKA_TOPIC, new
>>>             SimpleStringSchema(), props))
>>>
>>>             Then on kafkamanager -> consumers I have the groupID prod.
>>>
>>>             Hope it helps.
>>>
>>>             Best Regards,
>>>
>>>             Daniel Santos
>>>
>>>             On 11/08/2016 08:45 AM, vinay patil wrote:
>>>>             Hi Limbo,
>>>>
>>>>             I am using 0.9, I am not able to see updated results
>>>>             even after refreshing.
>>>>             There is some property that we have to set in order to
>>>>             make this work
>>>>
>>>>             Regards,
>>>>             Vinay Patil
>>>>
>>>>             On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache
>>>>             Flink User Mailing List archive.] <[hidden email]
>>>>             <http:///user/SendEmail.jtp?type=node&node=9969&i=0>>
>>>>             wrote:
>>>>
>>>>                 I am using kafka 0.8, just refresh the page and you
>>>>                 will see the updated results, it’s not auto update.
>>>>
>>>>                 There is our logstash consumer:
>>>>
>>>>
>>>>
>>>>                 and this is our flink consumer:
>>>>
>>>>
>>>>                 I find the flink consumer just write the offset of
>>>>                 the kafka partition to zookeeper without owner and
>>>>                 ids,
>>>>                 so we can’t find the consumer in the manager page,
>>>>                 we can only find the offset info.
>>>>
>>>>>                 在 2016年11月8日,下午12:02,vinay patil <[hidden
>>>>>                 email]
>>>>>                 <http:///user/SendEmail.jtp?type=node&node=9965&i=0>>
>>>>>                 写道:
>>>>>
>>>>>                 Hi Limbo,
>>>>>
>>>>>                 I can see the lag by using that URL, but the Lag
>>>>>                 there is not showing updated results, it does not
>>>>>                 change, also if you try to to change the consumer
>>>>>                 group value it will still show you the same value
>>>>>                 instead of saying consumer group does not exist or
>>>>>                 similar kind of error :)
>>>>>
>>>>>                 According to documentation of 0.9.x the offsets
>>>>>                 are stored in Kafka, but we can set offset.storage
>>>>>                 property to zookeeper.
>>>>>                 Even by setting this I am not getting the consumer
>>>>>                 listed.
>>>>>
>>>>>                 Kafka cli command also does not show this consumer
>>>>>
>>>>>                 Regards,
>>>>>                 Vinay Patil
>>>>>
>>>>>                 On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache
>>>>>                 Flink User Mailing List archive.]<<a
>>>>>                 href="x-msg://4/user/SendEmail.jtp?type=node&amp;node=9964&amp;i=0"
>>>>>                 target="_top" rel="nofollow" link="external"
>>>>>                 class="">[hidden email]>wrote:
>>>>>
>>>>>                     Hi,
>>>>>
>>>>>                     I have the same problem, I think the reason is
>>>>>                     that the consumer of flink use the low level API,
>>>>>                     and when I type the group name in manager url
>>>>>                     I can get the lag of the flink consumer, like
>>>>>                     this:
>>>>>
>>>>>                     http://your_manager_url/clusters/<kafka_name>/consumers/<consumer_name>/topic/<topic_name>/type/ZK
>>>>>                     <http://your_manager_url/clusters/%3Ckafka_name%3E/consumers/%3Cconsumer_name%3E/topic/%3Ctopic_name%3E/type/ZK>
>>>>>
>>>>>>                     在 2016年11月8日,上午5:12,Daniel Santos
>>>>>>                     <[hidden email]
>>>>>>                     <http://user/SendEmail.jtp?type=node&node=9962&i=0>>
>>>>>>                     写道:
>>>>>>
>>>>>>                     Hello,
>>>>>>
>>>>>>                     I have been using that setup.
>>>>>>                     From my understanding, if one desires to see
>>>>>>                     the offset being consumed by Flink on
>>>>>>                     KafkaManger, one has to set it up with
>>>>>>                     zookeeper. On 0.9 it will only serve as a
>>>>>>                     view of progress.
>>>>>>
>>>>>>                     Basically what's mandatory on 0.8 is optional
>>>>>>                     on 0.9, and for viewing purposes only.
>>>>>>
>>>>>>                     Best Regards,
>>>>>>                     Daniel Santos
>>>>>>
>>>>>>                     On November 7, 2016 7:13:54 PM GMT+00:00,
>>>>>>                     Vinay Patil <[hidden email]
>>>>>>                     <http://user/SendEmail.jtp?type=node&node=9962&i=1>>
>>>>>>                     wrote:
>>>>>>
>>>>>>                         Hi,
>>>>>>
>>>>>>                         I am monitoring Kafka using KafkaManager
>>>>>>                         for checking offset lag and other Kafka
>>>>>>                         metrics, however I am not able to see the
>>>>>>                          consumers when I use FlinkKafkaConsumer
>>>>>>                         , for console-consumer it shows them in
>>>>>>                         the Consumers list.
>>>>>>
>>>>>>                         I have set the required parameters for
>>>>>>                         the kafka consumer while running the
>>>>>>                         application.
>>>>>>
>>>>>>                         Has anyone faced this issue ?
>>>>>>                         I am using Kafka 0.9.0.1
>>>>>>
>>>>>>                         Regards,
>>>>>>                         Vinay Patil
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                     ------------------------------------------------------------------------
>>>>>                     If you reply to this email, your message will
>>>>>                     be added to the discussion below:
>>>>>                     http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html
>>>>>                     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html>
>>>>>                     To start a new topic under Apache Flink User
>>>>>                     Mailing List archive., email<a
>>>>>                     href="x-msg://4/user/SendEmail.jtp?type=node&amp;node=9964&amp;i=1"
>>>>>                     target="_top" rel="nofollow" link="external"
>>>>>                     class="">[hidden email]
>>>>>                     To unsubscribe from Apache Flink User Mailing
>>>>>                     List archive.,click here.
>>>>>                     NAML
>>>>>                     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>
>>>>>
>>>>>
>>>>>                 ------------------------------------------------------------------------
>>>>>                 View this message in context:Re: Kafka Monitoring
>>>>>                 <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
>>>>>                 Sent from theApache Flink User Mailing List
>>>>>                 archive. mailing list archive
>>>>>                 <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>atNabble.com
>>>>>                 <http://nabble.com/>.
>>>>
>>>>
>>>>
>>>>                 ------------------------------------------------------------------------
>>>>                 If you reply to this email, your message will be
>>>>                 added to the discussion below:
>>>>                 http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html
>>>>                 <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html>
>>>>
>>>>                 To start a new topic under Apache Flink User
>>>>                 Mailing List archive., email [hidden email]
>>>>                 <http:///user/SendEmail.jtp?type=node&node=9969&i=1>
>>>>                 To unsubscribe from Apache Flink User Mailing List
>>>>                 archive., click here.
>>>>                 NAML
>>>>                 <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>>
>>>>
>>>>
>>>>             ------------------------------------------------------------------------
>>>>             View this message in context: Re: Kafka Monitoring
>>>>             <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
>>>>             Sent from the Apache Flink User Mailing List archive.
>>>>             mailing list archive
>>>>             <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>>>             at Nabble.com.
>>>
>>>
>>>
>>>             ------------------------------------------------------------------------
>>>             If you reply to this email, your message will be added
>>>             to the discussion below:
>>>             http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9976.html
>>>             <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9976.html>
>>>
>>>             To start a new topic under Apache Flink User Mailing
>>>             List archive., email [hidden email]
>>>             <http:///user/SendEmail.jtp?type=node&node=9978&i=1>
>>>             To unsubscribe from Apache Flink User Mailing List
>>>             archive., click here.
>>>             NAML
>>>             <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>>
>>>         ------------------------------------------------------------------------
>>>         View this message in context: Re: Kafka Monitoring
>>>         <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html>
>>>         Sent from the Apache Flink User Mailing List archive.
>>>         mailing list archive
>>>         <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>>         at Nabble.com.
>>
>>
>>
>>         ------------------------------------------------------------------------
>>         If you reply to this email, your message will be added to the
>>         discussion below:
>>         http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9979.html
>>         <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9979.html>
>>
>>         To start a new topic under Apache Flink User Mailing List
>>         archive., email [hidden email]
>>         <http:///user/SendEmail.jtp?type=node&node=9980&i=1>
>>         To unsubscribe from Apache Flink User Mailing List archive.,
>>         click here.
>>         NAML
>>         <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>>
>>     ------------------------------------------------------------------------
>>     View this message in context: Re: Kafka Monitoring
>>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9980.html>
>>     Sent from the Apache Flink User Mailing List archive. mailing
>>     list archive
>>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>     at Nabble.com.
>
>
>
>     ------------------------------------------------------------------------
>     If you reply to this email, your message will be added to the
>     discussion below:
>     http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9982.html
>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9982.html>
>
>     To start a new topic under Apache Flink User Mailing List
>     archive., email [hidden email]
>     </user/SendEmail.jtp?type=node&node=9983&i=1>
>     To unsubscribe from Apache Flink User Mailing List archive., click
>     here.
>     NAML
>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> ------------------------------------------------------------------------
> View this message in context: Re: Kafka Monitoring 
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9983.html>
> Sent from the Apache Flink User Mailing List archive. mailing list 
> archive 
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> 
> at Nabble.com.


Mime
View raw message