nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: NIFI-2680
Date Fri, 09 Sep 2016 13:46:23 GMT
Chris,

I was looking into this, but have a few questions for you.

In NIFI-2680, it indicates that the problem was with PutKafka. This email seems to indicate
that you're seeing a problem with PublishKafka.
So just want to ensure that we're on the same page - are you seeing issues in both PutKafka
and PublishKafka?

I have been testing with PutKafka and putting a 5 MB FlowFile does indeed fail with an error
message saying that the FlowFile is too large.
It appears that this was fixed in NIFI-2445, which has gone into 0.8.0 and 1.0.0. However,
NIFI-2680 indicates that the problem affects 0.7.0,
0.8.0, and 1.0.0.

From your stack trace here it appears that you're using Kafka Processors from an 0.7.x release.
Have you verified that this issue still exists
with a newer version of NiFi?

Thanks!
-Mark



> On Sep 8, 2016, at 8:00 PM, McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote) <chris.mcdermott@hpe.com>
wrote:
> 
> Hello dev folks.
>  
> Would it be possible for someone to take a look at NIFI-2680?  https://issues.apache.org/jira/browse/NIFI-2680?jql=project%20%3D%20NIFI%20AND%20reporter%20%3D%20currentUser()
<https://issues.apache.org/jira/browse/NIFI-2680?jql=project%20%3D%20NIFI%20AND%20reporter%20%3D%20currentUser()>
> I believe I have a similar situation where writing to Kafka fails when the Kafka broker
is restarted.  The flowfile never seems to make it to Kafka but it transferred to the PutKafka
success output anyway. i.e. this is a data-loss bug.
>  
> Here is the traceback from the failure.
>  
> 2016-09-08 18:41:44,507 ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.kafka.KafkaPublisher
Failed while waiting for acks from Kafka
> java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NotLeaderForPartitionException:
This server is not the leader for that topic-partition.
>        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
~[kafka-clients-0.8.2.2.jar:na]
>        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:51)
~[kafka-clients-0.8.2.2.jar:na]
>        at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
~[kafka-clients-0.8.2.2.jar:na]
>        at org.apache.nifi.processors.kafka.KafkaPublisher.processAcks(KafkaPublisher.java:197)
[nifi-kafka-processors-oculus-0.7.x-SNAPSHOT.jar:oculus-0.7.x-SNAPSHOT]
>        at org.apache.nifi.processors.kafka.KafkaPublisher.publish(KafkaPublisher.java:140)
[nifi-kafka-processors-oculus-0.7.x-SNAPSHOT.jar:oculus-0.7.x-SNAPSHOT]
>        at org.apache.nifi.processors.kafka.PutKafka$1.process(PutKafka.java:315) [nifi-kafka-processors-oculus-0.7.x-SNAPSHOT.jar:oculus-0.7.x-SNAPSHOT]
>        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851)
[nifi-framework-core-oculus-0.7.x-SNAPSHOT.jar:oculus-0.7.x-SNAPSHOT]
>        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1822)
[nifi-framework-core-oculus-0.7.x-SNAPSHOT.jar:oculus-0.7.x-SNAPSHOT]
>        at org.apache.nifi.processors.kafka.PutKafka.doRendezvousWithKafka(PutKafka.java:311)
[nifi-kafka-processors-oculus-0.7.x-SNAPSHOT.jar:oculus-0.7.x-SNAPSHOT]
>        at org.apache.nifi.processors.kafka.PutKafka.rendezvousWithKafka(PutKafka.java:287)
[nifi-kafka-processors-oculus-0.7.x-SNAPSHOT.jar:oculus-0.7.x-SNAPSHOT]
>        at org.apache.nifi.processors.kafka.AbstractKafkaProcessor.onTrigger(AbstractKafkaProcessor.java:76)
[nifi-kafka-processors-oculus-0.7.x-SNAPSHOT.jar:oculus-0.7.x-SNAPSHOT]
>        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
[nifi-framework-core-oculus-0.7.x-SNAPSHOT.jar:oculus-0.7.x-SNAPSHOT]
>        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-oculus-0.7.x-SNAPSHOT.jar:oculus-0.7.x-SNAPSHOT]
>        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-oculus-0.7.x-SNAPSHOT.jar:oculus-0.7.x-SNAPSHOT]
>        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
[nifi-framework-core-oculus-0.7.x-SNAPSHOT.jar:oculus-0.7.x-SNAPSHOT]
>        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
>        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45]
>        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
>        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
>        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic-partition.
>  
> Thanks,
>  
> Chris McDermott
>  
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
>  
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message