nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: NiFi 1.3.0 Remote ports "disabled" after a while?
Date Thu, 15 Jun 2017 14:38:05 GMT
Joe,

Thanks for the info.

I think this might be similar to
https://issues.apache.org/jira/browse/NIFI-3900, but the fix didn't
account for RPGs.

I created this JIRA - https://issues.apache.org/jira/browse/NIFI-4075

-Bryan

On Thu, Jun 15, 2017 at 10:14 AM, Joe Gresock <jgresock@gmail.com> wrote:
> Btw, I was wrong about one statement in my last email.. this particular RPG
> (705cd4bd-1b98-3c4b-b467-c455ad52fd90) IS in the state where some nodes are
> STOPPED and some are RUNNING.  However, even on the nodes that show RUNNING
> in the flow.xml.gz, the queue is still blocked.
>
>
>
> On Thu, Jun 15, 2017 at 2:07 PM, Joe Gresock <jgresock@gmail.com> wrote:
>
>> Great questions, Koji:
>>
>> 1. RAW
>> 2. Push
>> 3. No, compression is not enabled
>> 4. Several, but it has the same effect to "Enable transmission" as it does
>> to specifically disable and then enable a specific port.  The key being
>> that all the ports are marked as enabled to start, even when not sending.
>> 5. This is anecdotal, but it feels like 1+ times per day.
>> 6. CentOS 6.9, Java 1.8.0_131.  This is a 4-node NiFi cluster.  There are
>> multiple self-RPGs, including this one, at various points on the graph.
>> 7. Attached.  The relevant queue that is inexplicably blocked shows this
>> in the log:
>> "Remote Process Group 705cd4bd-1b98-3c4b-b467-c455ad52fd90:
>> https://ip-172-31-55-35.ec2.internal:8443/nifi Thread-1" Id=258
>> TIMED_WAITING  on java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject@2edb0217
>>         at sun.misc.Unsafe.park(Native Method)
>>         at java.util.concurrent.locks.LockSupport.parkNanos(
>> LockSupport.java:215)
>>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
>> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
>>         at java.util.concurrent.ScheduledThreadPoolExecutor$
>> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
>>         at java.util.concurrent.ScheduledThreadPoolExecutor$
>> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
>>         at java.util.concurrent.ThreadPoolExecutor.getTask(
>> ThreadPoolExecutor.java:1067)
>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1127)
>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:617)
>>         at java.lang.Thread.run(Thread.java:748)
>>
>> 8. No proxy
>>
>> One strange thing I noticed was that the <scheduledState> is not
>> consistent in the flow.xml.gz between some of the nodes for other instances
>> of this self-RPG (but not for the one that is currently backing up):
>> 1. On disk, some of the nodes have this input port set to STOPPED, and
>> some have it set to RUNNING.
>> 2. In the UI, the input port is displayed as RUNNING.
>>
>>
>>
>> On Thu, Jun 15, 2017 at 12:21 AM, Koji Kawamura <ijokarumawak@gmail.com>
>> wrote:
>>
>>> Hi Joe,
>>>
>>> I haven't encountered that issue. Would you tell us more details? So
>>> that we can investigate it.
>>>
>>> 1. Which transport protocol are you using? RAW or HTTP
>>> 2. Is this a pull or push transfer? Pull = client NiFi pulls from
>>> remote OutputPort, Push = sends to remote InputPort
>>> 3. Is compression enabled?
>>> 4. How many ports are configured? If there are more than one, RPG
>>> indicates that it's 'Transmitting' as long as at least one port is
>>> transmitting, while others can be disabled. Right click a RPG then
>>> select 'Remote Ports' to see each port status.
>>> 5. How often does it happen? Stops once per hour/day/week??
>>> 6. Please share your environment, OS, Java version ... etc
>>> 7. If possible, please take stack-trace while you are facing that
>>> issue, and share it with us, by running: "$NIFI_HOME/bin/nifh.sh dump"
>>> stack-trace will be written to nifi-bootstrap.log
>>> 8. Is proxy used?
>>>
>>> Thanks,
>>> Koji
>>>
>>>
>>> On Thu, Jun 15, 2017 at 12:41 AM, Joe Gresock <jgresock@gmail.com> wrote:
>>> > Before I investigate further, has anyone encountered NiFi RPGs
>>> "disabling"
>>> > after a while in 1.3.0?  Their status is reported as
>>> enabled/transmitting,
>>> > but the flow files just sit there until I "enable transmission" again.
>>> > This happens for nearly all of the RPGs I have in my flow.  I just have
>>> to
>>> > keep "kick starting" them periodically to keep things flowing.
>>> >
>>> > This behavior did not happen in 1.2.0.
>>> > --
>>> > I know what it is to be in need, and I know what it is to have plenty.
>>> I
>>> > have learned the secret of being content in any and every situation,
>>> > whether well fed or hungry, whether living in plenty or in want.  I can
>>> do
>>> > all this through him who gives me strength.    *-Philippians 4:12-13*
>>>
>>
>>
>>
>> --
>> I know what it is to be in need, and I know what it is to have plenty.  I
>> have learned the secret of being content in any and every situation,
>> whether well fed or hungry, whether living in plenty or in want.  I can
>> do all this through him who gives me strength.    *-Philippians 4:12-13*
>>
>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.    *-Philippians 4:12-13*

Mime
View raw message