nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Carenza <nick.care...@thecontrolgroup.com>
Subject Flow seized, cannot list flow files for full queue
Date Wed, 21 Dec 2016 18:56:43 GMT
I am running into an issue where a processor will stop receiving flow files
from it's queue.

flow: tcp --(100,000)--> evaljsonpath  --(100,000)--> geoip
 --(100,000)--> putkafka

This time, putkafka is the processor that has stopped receiving flowfiles​

I will try to list the queue and I'll get a message that says the queue has
no flow files in it. I checked the http request and the response says there
are 100,000 flow files in the queue but the flowFileSummaries array is
empty.

GET
> /nifi-api/flowfile-queues/1d72b81f-0159-1000-d09b-dc33e81b35c2/listing-requests/22754339-0159-1000-2dc9-07db09366132
> HTTP/1.1
> {
>   "listingRequest": {
>     "id": "22754339-0159-1000-2dc9-07db09366132",
>     "uri": "
> http://ipaddress:8080/nifi-api/flowfile-queues/1d72b81f-0159-1000-d09b-dc33e81b35c2/listing-requests/22754339-0159-1000-2dc9-07db09366132
> ",
>     "submissionTime": "12/21/2016 17:37:07.385 UTC",
>     "lastUpdated": "17:37:07 UTC",
>     "percentCompleted": 100,
>     "finished": true,
>     "maxResults": 100,
>     "state": "Completed successfully",
>     "queueSize": {
>       "byteCount": 288609476,
>       "objectCount": 100000
>     },
>     "flowFileSummaries": [],
>     "sourceRunning": true,
>     "destinationRunning": true
>   }
> }


I tried stopping and starting all the processors, replacing the putkafka
with a new duplicate putkafka processor and moving the queue over to it,
restarting kafka itself. I ran a dump with all the processors "running".

Since this is not running in a production environment, as a last resort I
cleared the queue and then everything started flowing again.

I have experienced this issue many times since I have begun evaluating
Nifi. I have heard others having great success with it so I am convinced I
have misconfigured something. I have tried to provide any relevant
configuration information here:

# nifi.properties
nifi.version=1.1.0
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec
nifi.bored.yield.duration=10 millis
nifi.state.management.provider.local=local-provider
nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=1000
nifi.swap.in.period=5 sec
nifi.swap.in.threads=1
nifi.swap.out.period=5 sec
nifi.swap.out.threads=4
nifi.cluster.is.node=false
nifi.build.tag=nifi-1.1.0-RC2
nifi.build.branch=NIFI-3100-rc2
nifi.build.revision=f61e42c
nifi.build.timestamp=2016-11-26T04:39:37Z

# JVM memory settings
java.arg.2=-Xms28g
java.arg.3=-Xmx28g
java.arg.13=-XX:+UseG1GC

controller settings:
timer driven thread count: 10-20 (i have tried values from 10 to 20 and
still experience the issue)
event drive thread count: 5 (haven't touched)

processors:
concurrency: 1-20 (i have tried values from 1 to 20 and still experience
the issue)
scheduling: timer driven (run-schedule: 0 run-duration: 0)

queues:
backpressure flowfile count: 100,000
backpressure flowfile size: 1G

machine:
128g ram
20 cpu
disk: 3T

---

Really I have 2 questions:

1. Why is this happening?
2. Once the flow is in this state, how can I get it flowing again without
losing flowfiles?

Thanks,
Nick

Mime
View raw message