flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vinay patil <vinay18.pa...@gmail.com>
Subject Re: Kafka Monitoring
Date Wed, 09 Nov 2016 05:28:16 GMT
Hi Daniel,

Yes Now I am able to see it , this was just a dummy code I was running on
local VM.

However on cluster , I had enabled checkpointing, still I was not able to
see  the consumers, I guess I have to put the brokerPath as you have
provided in the properties.

>From where did you get the brokerPath ? I mean for which property have you
set it

Regards,
Vinay Patil

On Wed, Nov 9, 2016 at 12:32 AM, Daniel Santos [via Apache Flink User
Mailing List archive.] <ml-node+s2336050n9996h28@n4.nabble.com> wrote:

> Hello,
>
> On flink do you have the checkpoint enabled ?
>
> env.enableCheckpointing(interval = CHKPOINT_INTERVAL)
>
> Regards,
>
> Daniel Santos
>
> On 11/08/2016 12:30 PM, vinay patil wrote:
>
> Yes Kafka and Flink connect to that zookeeper only.
>
> Not sure why it is not listing the consumer
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 5:36 PM, Daniel Santos [via Apache Flink User
> Mailing List archive.] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9983&i=0>> wrote:
>
>> Hi,
>>
>> brokerPath is just optional.
>>
>> Used if you want to have multile kafka clusters.
>>
>> Each kafka cluster would connect to the same brokerPath.
>>
>> Since I have multiple clusters I use the brokerPath.
>>
>> From the looks of it you dont. So never mind it doesn't matter.
>>
>> You only have one zookeeper correct ?
>>
>> And kafka and flink connects to that only zookeeper ?
>>
>> Best Regards,
>>
>> Daniel Santos
>>
>> On 11/08/2016 11:18 AM, vinay patil wrote:
>>
>> Hi Daniel,
>>
>> Yes I have specified the zookeeper host in server.properties file , so
>> the broker is connected to zookeeper.
>>
>> https://kafka.apache.org/documentation#brokerconfigs  -> according to
>> this link, I guess all these configs are done in server.prop , so from
>> where did you get kafka09 as brokerPath ?
>>
>> this is my entry in server.prop file -> zookeeper.connect=localhost:2181
>> Have you set this as zkhost:2181/kafka09 ?
>>
>>
>> Regards,
>> Vinay Patil
>>
>> On Tue, Nov 8, 2016 at 4:27 PM, Daniel Santos [via Apache Flink User
>> Mailing List archive.] <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=9980&i=0>> wrote:
>>
>>> Hi,
>>>
>>> Your kafka broker is connected to zookeeper I believe.
>>>
>>> I am using kafka 0.9.0.1 my self too.
>>>
>>> On kafka broker 0.9.0.1 I have configured the zookeeper connect to a
>>> path, for instances :
>>>
>>> zk1:2181,zk2:2181,zk3:2181/kafka09
>>>
>>> https://kafka.apache.org/documentation#brokerconfigs
>>>
>>> Now on the flink side I would configure "props.setProperty("zookeeper.connect",
>>> zkHosts)" the same resulting in :
>>>
>>>
>>> props.setProperty("zookeeperconnect", "zk1:2181,zk2:2181,zk3:2181/ka
>>> fka09")
>>>
>>>
>>> That is what I mean by broker's path.
>>>
>>> Best Regards,
>>>
>>> Daniel Santos
>>>
>>> On 11/08/2016 10:49 AM, vinay patil wrote:
>>>
>>> Hi Daniel,
>>>
>>> I have the same properties set for the consumer and the same code
>>>
>>> *brokerspath only needed if you have set it on kafka config* -> I did
>>> not get this, do you mean to check the brokerspath in
>>> conf/server.properties file ? I have even tried by setting offset.storage
>>> property to zookeeeper, but still not getting the consumers listed
>>>
>>> I am using Kafka 0.9.0.1
>>>
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>> On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink User
>>> Mailing List archive.] <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9978&i=0>> wrote:
>>>
>>>> Hello,
>>>>
>>>> This is my config.
>>>>
>>>> On kafka props :
>>>>
>>>>     val props = new Properties()
>>>>
>>>>     props.setProperty("zookeeper.connect", zkHosts)
>>>>     props.setProperty("bootstrap.servers", kafHosts)
>>>>     props.setProperty("group.id", "prod")
>>>>     props.setProperty("auto.offset.reset", "earliest")
>>>>
>>>> Now for zkHosts beware that all your hosts quorum has to be included.
>>>>
>>>> For instances you have zk1 and zk2 and zk3 to form a quorum.
>>>>
>>>> Then it will result in zkHosts being -> zk1:2181,zk2:2181,zk3:2181/[brokerspath]
>>>> .
>>>>
>>>> brokerspath only needed if you have set it on kafka config. Ignore it
>>>> otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" .
>>>>
>>>> After that -> val source = env.addSource(new
>>>> FlinkKafkaConsumer09[String](KAFKA_TOPIC, new SimpleStringSchema(),
>>>> props))
>>>>
>>>> Then on kafkamanager -> consumers I have the groupID prod.
>>>>
>>>> Hope it helps.
>>>>
>>>> Best Regards,
>>>>
>>>> Daniel Santos
>>>> On 11/08/2016 08:45 AM, vinay patil wrote:
>>>>
>>>> Hi Limbo,
>>>>
>>>> I am using 0.9, I am not able to see updated results even after
>>>> refreshing.
>>>> There is some property that we have to set in order to make this work
>>>>
>>>> Regards,
>>>> Vinay Patil
>>>>
>>>> On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing
>>>> List archive.] <[hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=0>> wrote:
>>>>
>>>>> I am using kafka 0.8, just refresh the page and you will see the
>>>>> updated results, it’s not auto update.
>>>>>
>>>>> There is our logstash consumer:
>>>>>
>>>>>
>>>>>
>>>>> and this is our flink consumer:
>>>>>
>>>>>
>>>>> I find the flink consumer just write the offset of the kafka partition
>>>>> to zookeeper without owner and ids,
>>>>> so we can’t find the consumer in the manager page, we can only find
>>>>> the offset info.
>>>>>
>>>>> 在 2016年11月8日,下午12:02,vinay patil <[hidden email]
>>>>> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>>
写道:
>>>>>
>>>>> Hi Limbo,
>>>>>
>>>>> I can see the lag by using that URL, but the Lag there is not showing
>>>>> updated results, it does not change, also if you try to to change the
>>>>> consumer group value it will still show you the same value instead of
>>>>> saying consumer group does not exist or similar kind of error :)
>>>>>
>>>>> According to documentation of 0.9.x the offsets are stored in Kafka,
>>>>> but we can set offset.storage property to zookeeper.
>>>>> Even by setting this I am not getting the consumer listed.
>>>>>
>>>>> Kafka cli command also does not show this consumer
>>>>>
>>>>> Regards,
>>>>> Vinay Patil
>>>>>
>>>>> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User Mailing
>>>>> List archive.] <<a href="x-msg://4/user/SendEmail
>>>>> .jtp?type=node&amp;node=9964&amp;i=0" target="_top" rel="nofollow"
>>>>> link="external" class="">[hidden email]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have the same problem, I think the reason is that the consumer
of
>>>>>> flink use the low level API,
>>>>>> and when I type the group name in manager url I can get the lag of
>>>>>> the flink consumer, like this:
>>>>>>
>>>>>> http://your_manager_url/clusters/<kafka_name>/consumers/<con
>>>>>> sumer_name>/topic/<topic_name>/type/ZK
>>>>>>
>>>>>> 在 2016年11月8日,上午5:12,Daniel Santos <[hidden email]
>>>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>>
写道:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I have been using that setup.
>>>>>> From my understanding, if one desires to see the offset being
>>>>>> consumed by Flink on KafkaManger, one has to set it up with zookeeper.
On
>>>>>> 0.9 it will only serve as a view of progress.
>>>>>>
>>>>>> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing
>>>>>> purposes only.
>>>>>>
>>>>>> Best Regards,
>>>>>> Daniel Santos
>>>>>>
>>>>>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden
email]
>>>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>>
wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am monitoring Kafka using KafkaManager for checking offset
lag and
>>>>>>> other Kafka metrics, however I am not able to see the  consumers
when I use
>>>>>>> FlinkKafkaConsumer , for console-consumer it shows them in the
Consumers
>>>>>>> list.
>>>>>>>
>>>>>>> I have set the required parameters for the kafka consumer while
>>>>>>> running the application.
>>>>>>>
>>>>>>> Has anyone faced this issue ?
>>>>>>> I am using Kafka 0.9.0.1
>>>>>>>
>>>>>>> Regards,
>>>>>>> Vinay Patil
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>> If you reply to this email, your message will be added to the
>>>>>> discussion below:
>>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>>>> ble.com/Kafka-Monitoring-tp9957p9962.html
>>>>>> To start a new topic under Apache Flink User Mailing List archive.,
>>>>>> email <a href="x-msg://4/user/SendEmail.jtp?type=node&amp;node=9964&amp;i=1"
>>>>>> target="_top" rel="nofollow" link="external" class="">[hidden
email]
>>>>>> To unsubscribe from Apache Flink User Mailing List archive., click
>>>>>> here.
>>>>>> NAML
>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> View this message in context: Re: Kafka Monitoring
>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
>>>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>>>> archive
>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>>>>  at Nabble.com <http://nabble.com/>.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> If you reply to this email, your message will be added to the
>>>>> discussion below:
>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>>> ble.com/Kafka-Monitoring-tp9957p9965.html
>>>>> To start a new topic under Apache Flink User Mailing List archive.,
>>>>> email [hidden email]
>>>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=1>
>>>>> To unsubscribe from Apache Flink User Mailing List archive., click
>>>>> here.
>>>>> NAML
>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> View this message in context: Re: Kafka Monitoring
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
>>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>>> archive
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>>> at Nabble.com.
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> If you reply to this email, your message will be added to the
>>>> discussion below:
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>> ble.com/Kafka-Monitoring-tp9957p9976.html
>>>> To start a new topic under Apache Flink User Mailing List archive.,
>>>> email [hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=9978&i=1>
>>>> To unsubscribe from Apache Flink User Mailing List archive., click here
>>>> .
>>>> NAML
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>
>>>
>>> ------------------------------
>>> View this message in context: Re: Kafka Monitoring
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html>
>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>> at Nabble.com.
>>>
>>>
>>>
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>> ble.com/Kafka-Monitoring-tp9957p9979.html
>>> To start a new topic under Apache Flink User Mailing List archive.,
>>> email [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9980&i=1>
>>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>>> NAML
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>
>>
>> ------------------------------
>> View this message in context: Re: Kafka Monitoring
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9980.html>
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>> at Nabble.com.
>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Kafka-Monitoring-tp9957p9982.html
>> To start a new topic under Apache Flink User Mailing List archive., email [hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=9983&i=1>
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Kafka Monitoring
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9983.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-
> Monitoring-tp9957p9996.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p10006.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Mime
View raw message