nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Hamm <ceham...@gmail.com>
Subject Re: Queue wasn't emptying
Date Thu, 12 Nov 2015 13:25:42 GMT
that isn't clearing them out really. it gradually decreases but doesn't
clear out 500k items.
On Nov 12, 2015 8:22 AM, "Juan Sequeiros" <juan.sequeiros@onyxpoint.com>
wrote:

> To answer Christopher,
>
> You can clear out flowfiles on a queue by setting expiration on
> relationship line.
>
> On Thu, Nov 12, 2015 at 7:53 AM, Christopher Hamm <cehamm01@gmail.com>
> wrote:
>
>> I will look, but how do I just kill it with fire. What can I delete?
>> On Nov 12, 2015 7:42 AM, "Mark Payne" <markap14@hotmail.com> wrote:
>>
>>> Chris / Elli,
>>>
>>> What version of NiFi are you running? I am wondering if you may have
>>> been bitten by
>>> this bug [1]. This should affect only the 'master' branch, not any
>>> formally released
>>> versions of NiFi. If you built the application from source code, rather
>>> than using an
>>> already-released version, this may well be what is happening.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/NIFI-1155
>>>
>>>
>>> On Nov 12, 2015, at 2:32 AM, Christopher Hamm <email@christopherhamm.com>
>>> wrote:
>>>
>>> where does it dump? I cant get mine to expire fast enough. It is dev
>>> test I want to blow away but cant delete.
>>>
>>> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <joe.witt@gmail.com> wrote:
>>>
>>>> Elli,
>>>>
>>>> Can you share your flow configuration that we could possibly use to
>>>> replicate this?  Perhaps turn this into a JIRA and attach that.
>>>>
>>>> Also anytime you see something that appears 'stuck' please try to get
>>>> a stack dump (bin/nifi.sh dump)  As if there is truly a stuck thread
>>>> we'll see it there and what it is blocked on.
>>>>
>>>> Thanks
>>>> Joe
>>>>
>>>> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <
>>>> eliezer_schwarz@yahoo.com> wrote:
>>>> > I had a queue that built up overnight with several thousand
>>>> flowfiles. The
>>>> > queue was pointing to a RouteOnAttribute processor, which was
>>>> running. For
>>>> > some reason, the RouteOnAttribute processor wasn't emptying the queue
>>>> so the
>>>> > queue just built up. Stopping and starting the processor didn't help.
>>>> > However, simply restarting Nifi got the flow moving again. I can't
>>>> figure
>>>> > out why the queue was stuck.
>>>> >
>>>> > What could cause a queue to build up like that? How would restarting
>>>> Nifi
>>>> > get it going again? There weren't any errors in the log.  I did have
>>>> > backpressure on the relationship to the processor after the
>>>> RouteOnAttribute
>>>> > set to 1, but that queue was empty. There was no backpressure on the
>>>> queue
>>>> > heading in to the RouteOnAttribute processor. The RouteOnAttribute
>>>> processor
>>>> > has a Penalty Duration and a Yield of 1 sec. The relationships all
>>>> used the
>>>> > PriorityAttributeAnalyzer as the prioritizer.
>>>> >
>>>> > The interesting thing is I had the exact same problem on two
>>>> different nifis
>>>> > (one was version 0.2.0 and one 0.3.0). The same type of flow files
>>>> were
>>>> > going from one to the other using site-to-site (I was doing this for
>>>> load
>>>> > balancing). On the other Nifi, instead of restarting, I added another
>>>> > RouteOnAttribute processor and rerouted the stuck queue to the new one
>>>> > instead, and that also got things moving again. (I have a
>>>> backpressure since
>>>> > the processor after the RouteOnAttribute is a ControlRate processor,
>>>> and I
>>>> > only want to allow 1 flowfile through per second so as not to
>>>> overwhelm the
>>>> > system downstream. The backpressure is to assist with load balancing,
>>>> so if
>>>> > one pathway fills up, I have the other pathway which uses the
>>>> site-to-site
>>>> > to route the flow to the other nifi).
>>>> >
>>>> > Is there some combination of backpressure, penalty or yield along with
>>>> > prioritization that could cause a kind of deadlock-like situation?
>>>> Any ideas
>>>> > as to how I can prevent this from occurring?
>>>> >
>>>> > Thanks!
>>>> > -Elli
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Sincerely,
>>> Chris Hamm
>>> (E) cehamm01@gmail.com
>>> (Twitter) http://twitter.com/webhamm
>>> (Linkedin) http://www.linkedin.com/in/chrishamm
>>>
>>>
>>>
>

Mime
View raw message