flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vinay patil <vinay18.pa...@gmail.com>
Subject Re: Kafka Monitoring
Date Tue, 08 Nov 2016 11:18:23 GMT
Hi Daniel,

Yes I have specified the zookeeper host in server.properties file , so the
broker is connected to zookeeper.

https://kafka.apache.org/documentation#brokerconfigs  -> according to this
link, I guess all these configs are done in server.prop , so from where did
you get kafka09 as brokerPath ?

this is my entry in server.prop file -> zookeeper.connect=localhost:2181
Have you set this as zkhost:2181/kafka09 ?


Regards,
Vinay Patil

On Tue, Nov 8, 2016 at 4:27 PM, Daniel Santos [via Apache Flink User
Mailing List archive.] <ml-node+s2336050n9979h1@n4.nabble.com> wrote:

> Hi,
>
> Your kafka broker is connected to zookeeper I believe.
>
> I am using kafka 0.9.0.1 my self too.
>
> On kafka broker 0.9.0.1 I have configured the zookeeper connect to a path,
> for instances :
>
> zk1:2181,zk2:2181,zk3:2181/kafka09
>
> https://kafka.apache.org/documentation#brokerconfigs
>
> Now on the flink side I would configure "props.setProperty("zookeeper.connect",
> zkHosts)" the same resulting in :
>
>
> props.setProperty("zookeeperconnect", "zk1:2181,zk2:2181,zk3:2181/
> kafka09")
>
>
> That is what I mean by broker's path.
>
> Best Regards,
>
> Daniel Santos
>
> On 11/08/2016 10:49 AM, vinay patil wrote:
>
> Hi Daniel,
>
> I have the same properties set for the consumer and the same code
>
> *brokerspath only needed if you have set it on kafka config* -> I did not
> get this, do you mean to check the brokerspath in conf/server.properties
> file ? I have even tried by setting offset.storage property to zookeeeper,
> but still not getting the consumers listed
>
> I am using Kafka 0.9.0.1
>
>
> Regards,
> Vinay Patil
>
> On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink User
> Mailing List archive.] <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=9978&i=0>> wrote:
>
>> Hello,
>>
>> This is my config.
>>
>> On kafka props :
>>
>>     val props = new Properties()
>>
>>     props.setProperty("zookeeper.connect", zkHosts)
>>     props.setProperty("bootstrap.servers", kafHosts)
>>     props.setProperty("group.id", "prod")
>>     props.setProperty("auto.offset.reset", "earliest")
>>
>> Now for zkHosts beware that all your hosts quorum has to be included.
>>
>> For instances you have zk1 and zk2 and zk3 to form a quorum.
>>
>> Then it will result in zkHosts being -> zk1:2181,zk2:2181,zk3:2181/[brokerspath]
>> .
>>
>> brokerspath only needed if you have set it on kafka config. Ignore it
>> otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" .
>>
>> After that -> val source = env.addSource(new
>> FlinkKafkaConsumer09[String](KAFKA_TOPIC, new SimpleStringSchema(),
>> props))
>>
>> Then on kafkamanager -> consumers I have the groupID prod.
>>
>> Hope it helps.
>>
>> Best Regards,
>>
>> Daniel Santos
>> On 11/08/2016 08:45 AM, vinay patil wrote:
>>
>> Hi Limbo,
>>
>> I am using 0.9, I am not able to see updated results even after
>> refreshing.
>> There is some property that we have to set in order to make this work
>>
>> Regards,
>> Vinay Patil
>>
>> On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing
>> List archive.] <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=9969&i=0>> wrote:
>>
>>> I am using kafka 0.8, just refresh the page and you will see the updated
>>> results, it’s not auto update.
>>>
>>> There is our logstash consumer:
>>>
>>>
>>>
>>> and this is our flink consumer:
>>>
>>>
>>> I find the flink consumer just write the offset of the kafka partition
>>> to zookeeper without owner and ids,
>>> so we can’t find the consumer in the manager page, we can only find the
>>> offset info.
>>>
>>> 在 2016年11月8日,下午12:02,vinay patil <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>> 写道:
>>>
>>> Hi Limbo,
>>>
>>> I can see the lag by using that URL, but the Lag there is not showing
>>> updated results, it does not change, also if you try to to change the
>>> consumer group value it will still show you the same value instead of
>>> saying consumer group does not exist or similar kind of error :)
>>>
>>> According to documentation of 0.9.x the offsets are stored in Kafka, but
>>> we can set offset.storage property to zookeeper.
>>> Even by setting this I am not getting the consumer listed.
>>>
>>> Kafka cli command also does not show this consumer
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User Mailing
>>> List archive.] <<a href="x-msg://4/user/SendEmail
>>> .jtp?type=node&amp;node=9964&amp;i=0" target="_top" rel="nofollow"
>>> link="external" class="">[hidden email]> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have the same problem, I think the reason is that the consumer of
>>>> flink use the low level API,
>>>> and when I type the group name in manager url I can get the lag of the
>>>> flink consumer, like this:
>>>>
>>>> http://your_manager_url/clusters/<kafka_name>/consumers/<con
>>>> sumer_name>/topic/<topic_name>/type/ZK
>>>>
>>>> 在 2016年11月8日,上午5:12,Daniel Santos <[hidden email]
>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>> 写道:
>>>>
>>>> Hello,
>>>>
>>>> I have been using that setup.
>>>> From my understanding, if one desires to see the offset being consumed
>>>> by Flink on KafkaManger, one has to set it up with zookeeper. On 0.9 it
>>>> will only serve as a view of progress.
>>>>
>>>> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing
>>>> purposes only.
>>>>
>>>> Best Regards,
>>>> Daniel Santos
>>>>
>>>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden email]
>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am monitoring Kafka using KafkaManager for checking offset lag and
>>>>> other Kafka metrics, however I am not able to see the  consumers when
I use
>>>>> FlinkKafkaConsumer , for console-consumer it shows them in the Consumers
>>>>> list.
>>>>>
>>>>> I have set the required parameters for the kafka consumer while
>>>>> running the application.
>>>>>
>>>>> Has anyone faced this issue ?
>>>>> I am using Kafka 0.9.0.1
>>>>>
>>>>> Regards,
>>>>> Vinay Patil
>>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> If you reply to this email, your message will be added to the
>>>> discussion below:
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>>> ble.com/Kafka-Monitoring-tp9957p9962.html
>>>> To start a new topic under Apache Flink User Mailing List archive.,
>>>> email <a href="x-msg://4/user/SendEmail.jtp?type=node&amp;node=9964&amp;i=1"
>>>> target="_top" rel="nofollow" link="external" class="">[hidden email]
>>>> To unsubscribe from Apache Flink User Mailing List archive., click here
>>>> .
>>>> NAML
>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>
>>>
>>> ------------------------------
>>> View this message in context: Re: Kafka Monitoring
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>>> at Nabble.com <http://nabble.com/>.
>>>
>>>
>>>
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab
>>> ble.com/Kafka-Monitoring-tp9957p9965.html
>>> To start a new topic under Apache Flink User Mailing List archive.,
>>> email [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=1>
>>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>>> NAML
>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>
>>
>> ------------------------------
>> View this message in context: Re: Kafka Monitoring
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
>> at Nabble.com.
>>
>>
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Kafka-Monitoring-tp9957p9976.html
>> To start a new topic under Apache Flink User Mailing List archive., email [hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=9978&i=1>
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Kafka Monitoring
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html>
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-
> Monitoring-tp9957p9979.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9980.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Mime
View raw message