nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Carenza <>
Subject Flow seized, cannot list flow files for full queue
Date Wed, 21 Dec 2016 18:56:43 GMT
I am running into an issue where a processor will stop receiving flow files
from it's queue.

flow: tcp --(100,000)--> evaljsonpath  --(100,000)--> geoip
 --(100,000)--> putkafka

This time, putkafka is the processor that has stopped receiving flowfiles​

I will try to list the queue and I'll get a message that says the queue has
no flow files in it. I checked the http request and the response says there
are 100,000 flow files in the queue but the flowFileSummaries array is

> /nifi-api/flowfile-queues/1d72b81f-0159-1000-d09b-dc33e81b35c2/listing-requests/22754339-0159-1000-2dc9-07db09366132
> HTTP/1.1
> {
>   "listingRequest": {
>     "id": "22754339-0159-1000-2dc9-07db09366132",
>     "uri": "
> http://ipaddress:8080/nifi-api/flowfile-queues/1d72b81f-0159-1000-d09b-dc33e81b35c2/listing-requests/22754339-0159-1000-2dc9-07db09366132
> ",
>     "submissionTime": "12/21/2016 17:37:07.385 UTC",
>     "lastUpdated": "17:37:07 UTC",
>     "percentCompleted": 100,
>     "finished": true,
>     "maxResults": 100,
>     "state": "Completed successfully",
>     "queueSize": {
>       "byteCount": 288609476,
>       "objectCount": 100000
>     },
>     "flowFileSummaries": [],
>     "sourceRunning": true,
>     "destinationRunning": true
>   }
> }

I tried stopping and starting all the processors, replacing the putkafka
with a new duplicate putkafka processor and moving the queue over to it,
restarting kafka itself. I ran a dump with all the processors "running".

Since this is not running in a production environment, as a last resort I
cleared the queue and then everything started flowing again.

I have experienced this issue many times since I have begun evaluating
Nifi. I have heard others having great success with it so I am convinced I
have misconfigured something. I have tried to provide any relevant
configuration information here:

nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec
nifi.bored.yield.duration=10 millis
nifi.queue.swap.threshold=1000 sec
nifi.swap.out.period=5 sec

# JVM memory settings

controller settings:
timer driven thread count: 10-20 (i have tried values from 10 to 20 and
still experience the issue)
event drive thread count: 5 (haven't touched)

concurrency: 1-20 (i have tried values from 1 to 20 and still experience
the issue)
scheduling: timer driven (run-schedule: 0 run-duration: 0)

backpressure flowfile count: 100,000
backpressure flowfile size: 1G

128g ram
20 cpu
disk: 3T


Really I have 2 questions:

1. Why is this happening?
2. Once the flow is in this state, how can I get it flowing again without
losing flowfiles?


View raw message