Hi Qi,

I don't see anything suspicious in the log. Could you try out the https://github.com/gearpump/gearpump-java-example/tree/master/src/main/java/kafka2kafka example to see whether it's a framework bug ? 

"group-id" is set to "gearpump" if not configured by user. If you want to configure "group-id", you may create KafkaSource like 

Properties properties = new Properties();
properties.put("group-id", "my-group");
properties.put("zookeeper.servers", "localhost:2181");
KafkaSource source = new KafkaSource("topic", properties, storageFactory);


On Tue, May 3, 2016 at 3:13 PM 舒琦 <shuqi@eefung.com> wrote:

Hi Manu,


Could you also help me to check the log in the attachment.


How can I specify a group id when using Kafka Source, now I just set “group.id=XXX” in UserConfig.


Thanks.



————————
舒琦
地址:长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F
邮编:410013
电话:400-677-0986
传真:0731-88519609

 原始邮件 
发件人: 舒琦<shuqi@eefung.com>
发送时间: 2016年5月3日(周二) 14:19
主题: Re: Questions About Kafka Source

Hi Manu,


Gearpump: 0.7.6_2.11

Kafka: 0.8.2.1_2.10.


Thanks.


————————
舒琦
地址:长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F
邮编:410013
电话:400-677-0986
传真:0731-88519609

 原始邮件 
发件人: Manu Zhang<owenzhang1990@gmail.com>
发送时间: 2016年5月3日(周二) 14:13
主题: Re: Questions About Kafka Source

Hi Qi,

Your code looks right. Which gearpump version and kafka version have you used ?



On Tue, May 3, 2016 at 1:37 PM 舒琦 <shuqi@eefung.com> wrote:

Hi Manu,


Thanks for your help.


I used the kafka-console-consumer with the same zks and topic, and it can consume messages. There is still lots of messages in that topic.


Belowing is the function I used to get Kafka Soruce, could you please help to check if it is ok, thanks.




————————
舒琦
地址:长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F
邮编:410013
电话:400-677-0986
传真:0731-88519609

 原始邮件 
发件人: Manu Zhang<owenzhang1990@gmail.com>
发送时间: 2016年5月3日(周二) 12:40
主题: Re: Questions About Kafka Source

Hi Qi,

Neither the red ballon nor the message receive message throughput means any message has been consumed by KafkaSource. Those are messages source send to itself to trigger next Task execution. The metrics is a bit confusing and I think we need to fix this.

Yes, both zookeeper servers and kafka brokers configs are comma-separated list strings. One way to check whether your configurations is correct it to consume from the topic using kafka-console-consumer. This also makes sure the topic has data to consume.

Hope this helps.

Thanks,
Manu

On Tue, May 3, 2016 at 12:07 PM 舒琦 <shuqi@eefung.com> wrote:

Hi,


I constructed a DAG as show blowing, “kafka source”consumes messages from kafka topic “webs”, its metrics shows  that it consumes lots of messages, but actually there is no messages handled and I also can’t find active group under topic “webs”, the log is ok too.


I just wonder the properties of kafka for zks and brokers, if there is a list of zookeeper servers, should I use comma to separate? just like below:


zks=zk1:3181,zk2:3181,zk3:3181

brokers=kfk1:9096,kfk2:9096,kfk3:9096


Thanks for your help.




————————
Qi Shu