gearpump-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manu Zhang <owenzhang1...@gmail.com>
Subject Re: Questions About Kafka Source
Date Tue, 03 May 2016 08:55:58 GMT
Hi Qi,

I don't see anything suspicious in the log. Could you try out the
https://github.com/gearpump/gearpump-java-example/tree/master/src/main/java/kafka2kafka
example
to see whether it's a framework bug ?

"group-id" is set to "gearpump" if not configured by user. If you want to
configure "group-id", you may create KafkaSource like

*Properties properties = new Properties();*
*properties.put("group-id", "my-group");*
*properties.put("zookeeper.servers", "localhost:2181");*
*KafkaSource source = new KafkaSource("topic", properties, storageFactory);*


On Tue, May 3, 2016 at 3:13 PM 舒琦 <shuqi@eefung.com> wrote:

> Hi Manu,
>
>
> Could you also help me to check the log in the attachment.
>
>
> How can I specify a group id when using Kafka Source, now I just set “
> group.id=XXX” in UserConfig.
>
>
> Thanks.
>
>
> ————————
> 舒琦
> 地址:长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F
> 网址:http://www.eefung.com
> 微博:http://weibo.com/eefung
> 邮编:410013
> 电话:400-677-0986
> 传真:0731-88519609
>
>  原始邮件
> *发件人:* 舒琦<shuqi@eefung.com>
> *收件人:* user<user@gearpump.incubator.apache.org>
> *发送时间:* 2016年5月3日(周二) 14:19
> *主题:* Re: Questions About Kafka Source
>
> Hi Manu,
>
>
> Gearpump: 0.7.6_2.11
>
> Kafka: 0.8.2.1_2.10.
>
>
> Thanks.
>
> ————————
> 舒琦
> 地址:长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F
> 网址:http://www.eefung.com
> 微博:http://weibo.com/eefung
> 邮编:410013
> 电话:400-677-0986
> 传真:0731-88519609
>
>  原始邮件
> *发件人:* Manu Zhang<owenzhang1990@gmail.com>
> *收件人:* user<user@gearpump.incubator.apache.org>
> *发送时间:* 2016年5月3日(周二) 14:13
> *主题:* Re: Questions About Kafka Source
>
> Hi Qi,
>
> Your code looks right. Which gearpump version and kafka version have you
> used ?
>
>
>
> On Tue, May 3, 2016 at 1:37 PM 舒琦 <shuqi@eefung.com> wrote:
>
>> Hi Manu,
>>
>>
>> Thanks for your help.
>>
>>
>> I used the kafka-console-consumer with the same zks and topic, and it can
>> consume messages. There is still lots of messages in that topic.
>>
>>
>> Belowing is the function I used to get Kafka Soruce, could you please
>> help to check if it is ok, thanks.
>>
>>
>>
>>
>> ————————
>> 舒琦
>> 地址:长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F
>> 网址:http://www.eefung.com
>> 微博:http://weibo.com/eefung
>> 邮编:410013
>> 电话:400-677-0986
>> 传真:0731-88519609
>>
>>  原始邮件
>> *发件人:* Manu Zhang<owenzhang1990@gmail.com>
>> *收件人:* user<user@gearpump.incubator.apache.org>
>> *发送时间:* 2016年5月3日(周二) 12:40
>> *主题:* Re: Questions About Kafka Source
>>
>> Hi Qi,
>>
>> Neither the red ballon nor the message receive message throughput means
>> any message has been consumed by KafkaSource. Those are messages source
>> send to itself to trigger next Task execution. The metrics is a bit
>> confusing and I think we need to fix this.
>>
>> Yes, both zookeeper servers and kafka brokers configs are comma-separated
>> list strings. One way to check whether your configurations is correct it to
>> consume from the topic using kafka-console-consumer. This also makes sure
>> the topic has data to consume.
>>
>> Hope this helps.
>>
>> Thanks,
>> Manu
>>
>> On Tue, May 3, 2016 at 12:07 PM 舒琦 <shuqi@eefung.com> wrote:
>>
>>> Hi,
>>>
>>>
>>> I constructed a DAG as show blowing, “kafka source”consumes messages
>>> from kafka topic “webs”, its metrics shows  that it consumes lots of
>>> messages, but actually there is no messages handled and I also can’t find
>>> active group under topic “webs”, the log is ok too.
>>>
>>>
>>> I just wonder the properties of kafka for zks and brokers, if there is a
>>> list of zookeeper servers, should I use comma to separate? just like below:
>>>
>>>
>>> zks=zk1:3181,zk2:3181,zk3:3181
>>>
>>> brokers=kfk1:9096,kfk2:9096,kfk3:9096
>>>
>>>
>>> Thanks for your help.
>>>
>>>
>>>
>>>
>>> ————————
>>> Qi Shu
>>>
>>

Mime
View raw message