nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: Flow seized, cannot list flow files for full queue
Date Wed, 21 Dec 2016 19:19:57 GMT
Hey Nick,

Sorry to hear about these troubles. A couple of questions...

- What version of NiFi is this?
- What version of Kafka are you using?
- Which Kafka processor in NiFi are you using? It looks like PutKafka, but
just confirming.

Thanks,

Bryan


On Wed, Dec 21, 2016 at 2:00 PM, Nick Carenza <
nick.carenza@thecontrolgroup.com> wrote:

> I am running into an issue where a processor will stop receiving flow
> files from it's queue.
>
> flow: tcp --(100,000)--> evaljsonpath  --(100,000)--> geoip
>  --(100,000)--> putkafka
>
> This time, putkafka is the processor that has stopped receiving flowfiles​
>
> I will try to list the queue and I'll get a message that says the queue
> has no flow files in it. I checked the http request and the response says
> there are 100,000 flow files in the queue but the flowFileSummaries array
> is empty.
>
> GET /nifi-api/flowfile-queues/1d72b81f-0159-1000-d09b-dc33e81b35
>> c2/listing-requests/22754339-0159-1000-2dc9-07db09366132 HTTP/1.1
>> {
>>   "listingRequest": {
>>     "id": "22754339-0159-1000-2dc9-07db09366132",
>>     "uri": "http://ipaddress:8080/nifi-api/flowfile-queues/1d72b81f-015
>> 9-1000-d09b-dc33e81b35c2/listing-requests/22754339-0159-
>> 1000-2dc9-07db09366132",
>>     "submissionTime": "12/21/2016 17:37:07.385 UTC",
>>     "lastUpdated": "17:37:07 UTC",
>>     "percentCompleted": 100,
>>     "finished": true,
>>     "maxResults": 100,
>>     "state": "Completed successfully",
>>     "queueSize": {
>>       "byteCount": 288609476,
>>       "objectCount": 100000
>>     },
>>     "flowFileSummaries": [],
>>     "sourceRunning": true,
>>     "destinationRunning": true
>>   }
>> }
>
>
> I tried stopping and starting all the processors, replacing the putkafka
> with a new duplicate putkafka processor and moving the queue over to it,
> restarting kafka itself. I ran a dump with all the processors "running".
>
> Since this is not running in a production environment, as a last resort I
> cleared the queue and then everything started flowing again.
>
> I have experienced this issue many times since I have begun evaluating
> Nifi. I have heard others having great success with it so I am convinced I
> have misconfigured something. I have tried to provide any relevant
> configuration information here:
>
> # nifi.properties
> nifi.version=1.1.0
> nifi.flowcontroller.autoResumeState=true
> nifi.flowcontroller.graceful.shutdown.period=10 sec
> nifi.flowservice.writedelay.interval=500 ms
> nifi.administrative.yield.duration=30 sec
> nifi.bored.yield.duration=10 millis
> nifi.state.management.provider.local=local-provider
> nifi.swap.manager.implementation=org.apache.nifi.controller.
> FileSystemSwapManager
> nifi.queue.swap.threshold=1000
> nifi.swap.in.period=5 sec
> nifi.swap.in.threads=1
> nifi.swap.out.period=5 sec
> nifi.swap.out.threads=4
> nifi.cluster.is.node=false
> nifi.build.tag=nifi-1.1.0-RC2
> nifi.build.branch=NIFI-3100-rc2
> nifi.build.revision=f61e42c
> nifi.build.timestamp=2016-11-26T04:39:37Z
>
> # JVM memory settings
> java.arg.2=-Xms28g
> java.arg.3=-Xmx28g
> java.arg.13=-XX:+UseG1GC
>
> controller settings:
> timer driven thread count: 10-20 (i have tried values from 10 to 20 and
> still experience the issue)
> event drive thread count: 5 (haven't touched)
>
> processors:
> concurrency: 1-20 (i have tried values from 1 to 20 and still experience
> the issue)
> scheduling: timer driven (run-schedule: 0 run-duration: 0)
>
> queues:
> backpressure flowfile count: 100,000
> backpressure flowfile size: 1G
>
> machine:
> 128g ram
> 20 cpu
> disk: 3T
>
> ---
>
> Really I have 2 questions:
>
> 1. Why is this happening?
> 2. Once the flow is in this state, how can I get it flowing again without
> losing flowfiles?
>
> Thanks,
> Nick
>
>
>

Mime
View raw message