nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xmlk...@gmail.com
Subject Re: Nifi 0.50 and GetKafka Issues
Date Sun, 21 Feb 2016 23:46:24 GMT

If planing to support both, please make 0.9 version of getKafka not to depend on zookeeper,
as 0.9 Kafka consumes can directly communicate with brokers without needing explicit firewall
ports opened between NiFi and zookeeper 
Sumo

> On Feb 21, 2016, at 2:23 PM, Joe Witt <joe.witt@gmail.com> wrote:
> 
> Yeah the intent is to support 0.8 and 0.9.  Will figure something out.
> 
> Thanks
> Joe
> 
>> On Feb 21, 2016 4:47 PM, "West, Joshua" <Josh_West@bose.com> wrote:
>> Hi Oleg,
>> 
>> Hmm -- from what I can tell, this isn't a Zookeeper communication issue.  Nifi is
able to connect into the Kafka brokers' Zookeeper cluster and retrieve the list of the kafka
brokers to connect to.  Seems, from the logs, to be a problem when attempting to consume from
Kafka itself.
>> 
>> I'm guessing that the Kafka 0.9.0 client libraries just are not compatible with Kafka
0.8.2.1 so in order to use Nifi 0.5.0 with Kafka, the Kafka version must be >= 0.9.0.
>> 
>> Any change Nifi could add backwards compatible support for Kafka 0.8.2.1 too?  Let
you choose which client library version, when setting up the GetKafka processor?
>> 
>>  -- 
>> Josh West <josh_west@bose.com>
>> Bose Corporation
>> 
>> 
>>> On Sun, 2016-02-21 at 15:02 +0000, Oleg Zhurakousky wrote:
>>> Josh
>>> 
>>> Also, keep in mind that there are incompatible property names in Kafka between
the 0.7 and 0.8 releases. One of the change that went it was replacing “zk.connectiontimeout.ms”
with “zookeeper.connection.timeout.ms”. Not sure if it’s related though, but realizing
that 0.4.1 was relying on this property it’s value was completely ignored with 0.8 client
libraries (you could actually see the WARN message to that effect) and now it is not ignored,
so take a look and see if tinkering with its value changes something.
>>> 
>>> Cheers
>>> Oleg
>>>> On Feb 20, 2016, at 6:47 PM, Oleg Zhurakousky <ozhurakousky@hortonworks.com>
wrote:
>>>> 
>>>> Josh
>>>> 
>>>> The only change that ’s went and relevant to your issue is the fact that
we’ve upgraded client libraries to Kafka 0.9 and between 0.8 and 0.9 Kafka introduced wire
protocol changes that break compatibility.
>>>> I am still digging so stay tuned.
>>>> 
>>>> Oleg
>>>> 
>>>>> On Feb 20, 2016, at 4:10 PM, West, Joshua <Josh_West@bose.com>
wrote:
>>>>> 
>>>>> Hi Oleg and Joe,
>>>>> 
>>>>> Kafka 0.8.2.1
>>>>> 
>>>>> Attached is the app log with hostnames scrubbed.
>>>>> 
>>>>> Thanks for your help.  Much appreciated.
>>>>> 
>>>>>  -- 
>>>>> Josh West <josh_west@bose.com>
>>>>> Bose Corporation
>>>>> 
>>>>> 
>>>>>> On Sat, 2016-02-20 at 15:46 -0500, Joe Witt wrote:
>>>>>> And also what version of Kafka are you using?
>>>>>> 
>>>>>>> On Feb 20, 2016 3:37 PM, "Oleg Zhurakousky" <ozhurakousky@hortonworks.com>
wrote:
>>>>>>> Josh
>>>>>>> 
>>>>>>> Any chance to attache the app-log or relevant stack trace?
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Oleg
>>>>>>> 
>>>>>>>> On Feb 20, 2016, at 3:30 PM, West, Joshua <Josh_West@bose.com>
wrote:
>>>>>>>> 
>>>>>>>> Hi folks,
>>>>>>>> 
>>>>>>>> I've upgraded from Nifi 0.4.1 to 0.5.0 and I am no longer
able to use the GetKafka processor.  I'm seeing errors like so:
>>>>>>>> 
>>>>>>>> 2016-02-20 20:10:14,953 WARN [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0]
kafka.consumer.ConsumerFetcherThread [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0],
Error in fetchkafka.consumer.ConsumerFetcherThread$FetchRequest@7b49a642. Possible cause:
java.lang.IllegalArgumentException
>>>>>>>> 
>>>>>>>> ^ Note  the hostname of the server has been scrubbed.
>>>>>>>> 
>>>>>>>> My configuration is pretty generic, except that with Zookeeper
we use a different root path, so our Zookeeper connect string looks like so:
>>>>>>>> 
>>>>>>>> zookeeper-node1:2181,zookeeper-node2:2181,zookeeper-node3:2181/kafka
>>>>>>>> 
>>>>>>>> Is anybody else experiencing issues?
>>>>>>>> 
>>>>>>>> Thanks.
>>>>>>>> 
>>>>>>>>  -- 
>>>>>>>> Josh West <josh_west@bose.com>
>>>>>>>> Cloud Architect
>>>>>>>> Bose Corporation
>>>>>>>> 
>>>>> <nifi-app.log.kafkaissues.bz2>

Mime
View raw message