nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Psaltis <psaltis.and...@gmail.com>
Subject Re: Consumekafka error I
Date Thu, 14 Jun 2018 14:09:56 GMT
Hi Faisal,
How many partitions are there for that TEST_KAFKA_TOPIC topic?

On Thu, Jun 14, 2018 at 9:06 PM Faisal Durrani <te04.0172@gmail.com> wrote:

> Hi Mark, The heap size is set to 4 gb and the time driven threas count is
> set to 32 since its a 8 core  node( we have a cluster of 4 nifi nodes) .
> Despite of all this we keep getting these errors :( .
>
>
>
> On Thu, 14 Jun 2018, 10:02 p.m. Mark Payne, <markap14@hotmail.com> wrote:
>
>> Faisal,
>>
>> How much heap do you have allocated to your NiFi instance? In
>> conf/bootstrap.conf
>> the default value is 512 MB. If you haven't changed that, you could be
>> just running out of
>> heap.
>>
>> Also, have you changed the maximum number of threads available to your
>> NiFi instance?
>> In the top-right menu you can go to Controller Settings. The default for
>> "Maximum Timer Driven
>> Thread Count" is 10, but you'll definitely want to increase that for your
>> use case.
>>
>> Also, how many cores does the VM/node that NiFi is running on have?
>>
>> Thanks
>> -Mark
>>
>>
>> On Jun 14, 2018, at 3:39 AM, Faisal Durrani <te04.0172@gmail.com> wrote:
>>
>> it appears that the error only comes when i increase the number of
>> consumers. It seems to work fine with around 100 consumers but as soon as i
>> bump it up at about 150 consumers all hell breaks lose with error such as
>> below
>>
>> "Commit cannot be completed due to group rebalance"
>> "Failed to retain connection due to No current assignment for partition
>> TEST_KAFKA_TOPIC: "
>>
>> Quite strangely the consumers start consuming messages that have already
>> been read before. I cant wrap me head around why that is happening.
>> Can someone point to the right direction here? I tried wild carding all
>> the topics in one consumer but then it becomes very slow. All the consumers
>> are running on the primary node with single concurrent tasks .
>>
>> On Mon, Jun 11, 2018 at 11:17 AM Joe Witt <joe.witt@gmail.com> wrote:
>>
>>> So you have a unique instance of the ConsumeKafka proc for each topic
>>> rhen, right?
>>>
>>> Id increase the flow controller thread pool size by quite a bit as
>>> well.
>>>
>>> On Sun, Jun 10, 2018, 10:13 PM Faisal Durrani <te04.0172@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Yes the kafka service is hosted on a single server while NIFI is on a
>>>> cluster of 4 servers. I'm not entirely sure what wild carding of topics is
>>>> but kafka is integrated with a Oracle golden gate and the topics are auto
>>>> generated as soon as a new table is created in Oracle.
>>>>
>>>> "If you want that in a single instance you may need to alter the
>>>> timeout associated with any single kafka consumer. "
>>>> Can let me know which configuration is this?
>>>>
>>>> The consumer kafka processors are scheduled(Timer driven ) to run on
>>>> all four nodes with 4 concurrent task so i assume there will be 16
>>>> threads.( the Maximum timer driven thread count is set to 40)
>>>>
>>>> We tested the whole data flow with about 15-20 consumers and everything
>>>> worked fine with out any errors. We started getting all these wired errors
>>>> as soon as we bumped up our load test with 150+ consume kafka processors.
>>>>
>>>> On Mon, Jun 11, 2018 at 10:59 AM Joe Witt <joe.witt@gmail.com> wrote:
>>>>
>>>>> Hello
>>>>>
>>>>> Is this a single instance with wildcarding of topics?  Please share
>>>>> config details.
>>>>>
>>>>> If you want that in a single instance you may need to alter the
>>>>> timeout associated with any single kafka consumer.  The assignment will
be
>>>>> per topic per partion.  How many he threads for that processor?
>>>>>
>>>>> Finally, consider using ConsumeKafkaRecord and if you are using kaka
1
>>>>> or newer use the latest processor.
>>>>>
>>>>> thanks
>>>>>
>>>>> On Sun, Jun 10, 2018, 9:21 PM Faisal Durrani <te04.0172@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Does anyone know about this error from Kafka? I am using Nifi 1.5.0
>>>>>> with ConsumerKafka processor.
>>>>>>
>>>>>> ConsumeKafka[id=34753ed3-9dd6-15ed-9c91-147026236eee] Failed to
>>>>>> retain connection due to No current assignment for partition
>>>>>> TEST_KAFKA_TOPIC:
>>>>>>
>>>>>> This is the first time we are testing Nifi to consume from over 200
>>>>>> topics and its failing terribly so far. When this error goes the
other one
>>>>>> comes up which is as below
>>>>>>
>>>>>> Was interrupted while trying to communicate with Kafka with lease
>>>>>> org.apache.nifi.processors.kafka.pubsub.ConsumerPool$
>>>>>> SimpleConsumerLease@6cb8afba. Will roll back session and discard
any
>>>>>> partially received data.
>>>>>>
>>>>>
>>

Mime
View raw message