gearpump-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manu Zhang <owenzhang1...@gmail.com>
Subject Re: Questions About Kafka Source
Date Tue, 03 May 2016 06:13:21 GMT
Hi Qi,

Your code looks right. Which gearpump version and kafka version have you
used ?



On Tue, May 3, 2016 at 1:37 PM 舒琦 <shuqi@eefung.com> wrote:

> Hi Manu,
>
>
> Thanks for your help.
>
>
> I used the kafka-console-consumer with the same zks and topic, and it can
> consume messages. There is still lots of messages in that topic.
>
>
> Belowing is the function I used to get Kafka Soruce, could you please help
> to check if it is ok, thanks.
>
>
>
>
> ————————
> 舒琦
> 地址:长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F
> 网址:http://www.eefung.com
> 微博:http://weibo.com/eefung
> 邮编:410013
> 电话:400-677-0986
> 传真:0731-88519609
>
>  原始邮件
> *发件人:* Manu Zhang<owenzhang1990@gmail.com>
> *收件人:* user<user@gearpump.incubator.apache.org>
> *发送时间:* 2016年5月3日(周二) 12:40
> *主题:* Re: Questions About Kafka Source
>
> Hi Qi,
>
> Neither the red ballon nor the message receive message throughput means
> any message has been consumed by KafkaSource. Those are messages source
> send to itself to trigger next Task execution. The metrics is a bit
> confusing and I think we need to fix this.
>
> Yes, both zookeeper servers and kafka brokers configs are comma-separated
> list strings. One way to check whether your configurations is correct it to
> consume from the topic using kafka-console-consumer. This also makes sure
> the topic has data to consume.
>
> Hope this helps.
>
> Thanks,
> Manu
>
> On Tue, May 3, 2016 at 12:07 PM 舒琦 <shuqi@eefung.com> wrote:
>
>> Hi,
>>
>>
>> I constructed a DAG as show blowing, “kafka source”consumes messages from
>> kafka topic “webs”, its metrics shows  that it consumes lots of messages,
>> but actually there is no messages handled and I also can’t find active
>> group under topic “webs”, the log is ok too.
>>
>>
>> I just wonder the properties of kafka for zks and brokers, if there is a
>> list of zookeeper servers, should I use comma to separate? just like below:
>>
>>
>> zks=zk1:3181,zk2:3181,zk3:3181
>>
>> brokers=kfk1:9096,kfk2:9096,kfk3:9096
>>
>>
>> Thanks for your help.
>>
>>
>>
>>
>> ————————
>> Qi Shu
>>
>

Mime
View raw message