nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Dyer <michael.d...@trapezoid.com>
Subject Re: Nifi 0.50 and GetKafka Issues
Date Tue, 15 Mar 2016 17:21:19 GMT
Oleg,

It's part of the Cloudera distro.  Not sure of the lineage beyond that.
Here are a couple of links.

https://community.cloudera.com/t5/Data-Ingestion-Integration/New-Kafka-0-8-2-0-1-kafka1-3-1-p0-9-Parcel-What-are-the-changes/td-p/30506
http://archive.cloudera.com/kafka/parcels/1.3.1/

Michael


On Tue, Mar 15, 2016 at 12:01 PM, Oleg Zhurakousky <
ozhurakousky@hortonworks.com> wrote:

> Michael
>
> What is KAFKA-0.8.2.0-1.kafka1.3.1.p0.9? I mean where can I get that
> build?
> I guess based on the previous email we’ve tested our code with 3 versions
> of ASF distribution of Kafka and the above version tells me that it may be
> some kind of fork.
>
> Also, we are considering downgrading Kafka dependencies back to the 0.8
> and of 0.7 provide a new version of Kafka processors that utilize Kafka 0.9
> new producer/consumer API
>
> Thanks
> Oleg
>
> On Mar 15, 2016, at 11:46 AM, Michael Dyer <michael.dyer@trapezoid.com>
> wrote:
>
> Joe,
>
> I'm seeing a similar issue moving from 0.3.0 to 0.5.1 with
> KAFKA-0.8.2.0-1.kafka1.3.1.p0.9.
>
> I can see the tasks/time counter increment on the processor but no flow
> data ever leaves the processor.  There are no errors shown in the bulletin
> board.  The app log shows below (repeating).
>
> The rename 0.4.1 nar to 0.5.1 nar trick (restart nifi) works, except that
> the '*batch size*' value does not seem to be honored.  I have my batch
> size set to 10000, but I'm seeing files written continually (every few
> seconds) with much smaller sizes.  I suspect this has to do with the `
> *auto.offset.reset`* value which defaults to `*largest*`.  From what I
> have read 'smallest' causes the client to start at the beginning which
> sounds like I would be retrieving duplicates.
>
> Renaming 0.3.0 nar to 0.5.1 (restart nifi) restores the original behavior.
>
> yBuffer([[netflow5,0], initOffset 297426 to broker BrokerEndPoint(176,
> n2.foo.bar.com,9092)] )
> 2016-03-15 07:45:17,237 WARN
> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176]
> kafka.consumer.ConsumerFetcherThread
> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176],
> Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@11bd00f6.
> Possible cause: java.lang.IllegalArgumentException
> 2016-03-15 07:45:17,443 INFO
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread]
> kafka.utils.VerifiableProperties Verifying properties
> 2016-03-15 07:45:17,443 INFO
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread]
> kafka.utils.VerifiableProperties Property client.id is overridden to
> NiFi-b6c67ee3-aa9e-419d-8a57-84ab5e76c017
> 2016-03-15 07:45:17,443 INFO
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread]
> kafka.utils.VerifiableProperties Property metadata.broker.list is
> overridden to n3.foo.bar.com:9092,n2.foo.bar.com:9092,n4.foo.bar.com:9092
> 2016-03-15 07:45:17,443 INFO
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread]
> kafka.utils.VerifiableProperties Property request.timeout.ms is
> overridden to 30000
> 2016-03-15 07:45:17,443 INFO
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread]
> kafka.client.ClientUtils$ Fetching metadata from broker BrokerEndPoint(196,
> n4.foo.bar.com,9092) with correlation id 14 for 1 topic(s) Set(netflow5)
> 2016-03-15 07:45:17,443 INFO
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread]
> kafka.producer.SyncProducer Connected to n4.foo.bar.com:9092 for producing
> 2016-03-15 07:45:17,444 INFO
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread]
> kafka.producer.SyncProducer Disconnecting from n4.foo.bar.com:9092
> 2016-03-15 07:45:17,444 INFO
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread]
> kafka.consumer.ConsumerFetcherManager
> [ConsumerFetcherManager-1458053114395] Added fetcher for partitions
> ArrayBuffer([[netflow5,0], initOffset 297426 to broker BrokerEndPoint(176,
> n2.foo.bar.com,9092)] )
> 2016-03-15 07:45:17,449 WARN
> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176]
> kafka.consumer.ConsumerFetcherThread
> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176],
> Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@3e69ef73.
> Possible cause: java.lang.IllegalArgumentException
> 2016-03-15 07:45:17,626 INFO [NiFi Web Server-259]
> o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling
> GetKafka[id=4943a24e-af5c-4392-bc45-7008f30674bb] to run
> 2016-03-15 07:45:17,626 INFO [Timer-Driven Process Thread-3]
> k.consumer.ZookeeperConsumerConnector
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0],
> ZKConsumerConnector shutting down
> 2016-03-15 07:45:17,632 INFO [Timer-Driven Process Thread-3]
> kafka.consumer.ConsumerFetcherManager
> [ConsumerFetcherManager-1458053114395] Stopping leader finder thread
> 2016-03-15 07:45:17,633 INFO [Timer-Driven Process Thread-3]
> k.c.ConsumerFetcherManager$LeaderFinderThread
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread],
> Shutting down
> 2016-03-15 07:45:17,634 INFO
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread]
> k.c.ConsumerFetcherManager$LeaderFinderThread
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread],
> Stopped
> 2016-03-15 07:45:17,634 INFO [Timer-Driven Process Thread-3]
> k.c.ConsumerFetcherManager$LeaderFinderThread
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-leader-finder-thread],
> Shutdown completed
> 2016-03-15 07:45:17,634 INFO [Timer-Driven Process Thread-3]
> kafka.consumer.ConsumerFetcherManager
> [ConsumerFetcherManager-1458053114395] Stopping all fetchers
> 2016-03-15 07:45:17,634 INFO [Timer-Driven Process Thread-3]
> kafka.consumer.ConsumerFetcherThread
> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176],
> Shutting down
> 2016-03-15 07:45:17,634 INFO
> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176]
> kafka.consumer.ConsumerFetcherThread
> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176],
> Stopped
> 2016-03-15 07:45:17,635 INFO [Timer-Driven Process Thread-3]
> kafka.consumer.ConsumerFetcherThread
> [ConsumerFetcherThread-b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0-0-176],
> Shutdown completed
> 2016-03-15 07:45:17,635 INFO [Timer-Driven Process Thread-3]
> kafka.consumer.ConsumerFetcherManager
> [ConsumerFetcherManager-1458053114395] All connections stopped
> 2016-03-15 07:45:17,635 INFO [ZkClient-EventThread-302-192.168.1.1:2181]
> org.I0Itec.zkclient.ZkEventThread Terminate ZkClient event thread.
> 2016-03-15 07:45:17,638 INFO [Timer-Driven Process Thread-3]
> org.apache.zookeeper.ZooKeeper Session: 0x1535e2aa53b3f61 closed
> 2016-03-15 07:45:17,638 INFO [Timer-Driven Process Thread-3]
> k.consumer.ZookeeperConsumerConnector
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0],
> ZKConsumerConnector shutdown completed in 11 ms
> 2016-03-15 07:45:17,639 INFO [Timer-Driven Process Thread-4-EventThread]
> org.apache.zookeeper.ClientCnxn EventThread shut down
> 2016-03-15 07:45:17,745 INFO [Flow Service Tasks Thread-2]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controller.FlowController@45d22bd5 // Another save
> pending = false
> 2016-03-15 07:45:18,414 INFO
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0_watcher_executor]
> k.consumer.ZookeeperConsumerConnector
> [b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0],
> stopping watcher executor thread for consumer
> b6c67ee3-aa9e-419d-8a57-84ab5e76c017_nifi-1458053114390-d60f3cc0
>
> Hope this helps...
>
> Michael
>
> On Mon, Feb 22, 2016 at 4:12 PM, Joe Witt <joe.witt@gmail.com> wrote:
>
>> Sorry to clarify it working against all three of these at once:
>> - Kafka 0.8.1.1
>> - Kafka 0.8.2.1
>> - Kafka 0.9.0.1
>>
>> Thanks
>> Joe
>>
>> On Mon, Feb 22, 2016 at 4:12 PM, Joe Witt <joe.witt@gmail.com> wrote:
>> > All just as a point of reference we now have a live system running
>> > that is on NiFi 0.5.0 and feeding three versions of Kafka at once:
>> > - 0.8.1
>> > - 0.8.2.0
>> > - 0.9.0.1
>> >
>> > So perhaps there are some particular configurations that cause issues.
>> > Can you share more details about your configuration of Kafka/NiFi and
>> > what sort of security is enabled?
>> >
>> > Thanks
>> > Joe
>> >
>> > On Mon, Feb 22, 2016 at 1:01 PM, Kyle Burke <kyle.burke@ignitionone.com>
>> wrote:
>> >> I replaced my 0.5.0 kafka nar with the 0.4.1 kakfa nar and it fixed my
>> kafka
>> >> issue. I renamed the 0.4.1 nar to be 0.5.0.nar and restart nifi and my
>> kafka
>> >> processor started reading my 0.8.2.1 stream. Not elegant but glad it
>> worked.
>> >>
>> >>
>> >> Respectfully,
>> >>
>> >> Kyle Burke | Data Science Engineer
>> >> IgnitionOne - Marketing Technology. Simplified.
>> >> Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309
>> >> Direct: 404.961.3918
>> >>
>> >>
>> >> From: Joe Witt
>> >> Reply-To: "users@nifi.apache.org"
>> >> Date: Sunday, February 21, 2016 at 5:23 PM
>> >> To: "users@nifi.apache.org"
>> >> Subject: Re: Nifi 0.50 and GetKafka Issues
>> >>
>> >> Yeah the intent is to support 0.8 and 0.9.  Will figure something out.
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Feb 21, 2016 4:47 PM, "West, Joshua" <Josh_West@bose.com> wrote:
>> >>>
>> >>> Hi Oleg,
>> >>>
>> >>> Hmm -- from what I can tell, this isn't a Zookeeper communication
>> issue.
>> >>> Nifi is able to connect into the Kafka brokers' Zookeeper cluster and
>> >>> retrieve the list of the kafka brokers to connect to.  Seems, from
>> the logs,
>> >>> to be a problem when attempting to consume from Kafka itself.
>> >>>
>> >>> I'm guessing that the Kafka 0.9.0 client libraries just are not
>> compatible
>> >>> with Kafka 0.8.2.1 so in order to use Nifi 0.5.0 with Kafka, the Kafka
>> >>> version must be >= 0.9.0.
>> >>>
>> >>> Any change Nifi could add backwards compatible support for Kafka
>> 0.8.2.1
>> >>> too?  Let you choose which client library version, when setting up the
>> >>> GetKafka processor?
>> >>>
>> >>> --
>> >>> Josh West <josh_west@bose.com>
>> >>> Bose Corporation
>> >>>
>> >>>
>> >>> On Sun, 2016-02-21 at 15:02 +0000, Oleg Zhurakousky wrote:
>> >>>
>> >>> Josh
>> >>>
>> >>> Also, keep in mind that there are incompatible property names in Kafka
>> >>> between the 0.7 and 0.8 releases. One of the change that went it was
>> >>> replacing “zk.connectiontimeout.ms” with “
>> zookeeper.connection.timeout.ms”.
>> >>> Not sure if it’s related though, but realizing that 0.4.1 was relying
>> on
>> >>> this property it’s value was completely ignored with 0.8 client
>> libraries
>> >>> (you could actually see the WARN message to that effect) and now it
>> is not
>> >>> ignored, so take a look and see if tinkering with its value changes
>> >>> something.
>> >>>
>> >>> Cheers
>> >>> Oleg
>> >>>
>> >>> On Feb 20, 2016, at 6:47 PM, Oleg Zhurakousky
>> >>> <ozhurakousky@hortonworks.com> wrote:
>> >>>
>> >>> Josh
>> >>>
>> >>> The only change that ’s went and relevant to your issue is the fact
>> that
>> >>> we’ve upgraded client libraries to Kafka 0.9 and between 0.8 and 0.9
>> Kafka
>> >>> introduced wire protocol changes that break compatibility.
>> >>> I am still digging so stay tuned.
>> >>>
>> >>> Oleg
>> >>>
>> >>> On Feb 20, 2016, at 4:10 PM, West, Joshua <Josh_West@bose.com>
wrote:
>> >>>
>> >>> Hi Oleg and Joe,
>> >>>
>> >>> Kafka 0.8.2.1
>> >>>
>> >>> Attached is the app log with hostnames scrubbed.
>> >>>
>> >>> Thanks for your help.  Much appreciated.
>> >>>
>> >>> --
>> >>> Josh West <josh_west@bose.com>
>> >>> Bose Corporation
>> >>>
>> >>>
>> >>> On Sat, 2016-02-20 at 15:46 -0500, Joe Witt wrote:
>> >>>
>> >>> And also what version of Kafka are you using?
>> >>>
>> >>> On Feb 20, 2016 3:37 PM, "Oleg Zhurakousky" <
>> ozhurakousky@hortonworks.com>
>> >>> wrote:
>> >>>
>> >>> Josh
>> >>>
>> >>> Any chance to attache the app-log or relevant stack trace?
>> >>>
>> >>> Thanks
>> >>> Oleg
>> >>>
>> >>> On Feb 20, 2016, at 3:30 PM, West, Joshua <Josh_West@bose.com>
wrote:
>> >>>
>> >>> Hi folks,
>> >>>
>> >>> I've upgraded from Nifi 0.4.1 to 0.5.0 and I am no longer able to use
>> the
>> >>> GetKafka processor.  I'm seeing errors like so:
>> >>>
>> >>> 2016-02-20 20:10:14,953 WARN
>> >>>
>> [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0]
>> >>> kafka.consumer.ConsumerFetcherThread
>> >>>
>> [ConsumerFetcherThread-NiFi-sldjflkdsjflksjf_**SCRUBBED**-1455999008728-5b8c7108-0-0],
>> >>> Error in
>> fetchkafka.consumer.ConsumerFetcherThread$FetchRequest@7b49a642.
>> >>> Possible cause: java.lang.IllegalArgumentException
>> >>>
>> >>> ^ Note  the hostname of the server has been scrubbed.
>> >>>
>> >>> My configuration is pretty generic, except that with Zookeeper we use
>> a
>> >>> different root path, so our Zookeeper connect string looks like so:
>> >>>
>> >>> zookeeper-node1:2181,zookeeper-node2:2181,zookeeper-node3:2181/kafka
>> >>>
>> >>> Is anybody else experiencing issues?
>> >>>
>> >>> Thanks.
>> >>>
>> >>> --
>> >>> Josh West <josh_west@bose.com>
>> >>>
>> >>> Cloud Architect
>> >>> Bose Corporation
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> <nifi-app.log.kafkaissues.bz2>
>> >>>
>> >>>
>> >>>
>> >>
>>
>
>
>

Mime
View raw message