nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-2015) Corrupted flow file leads to a wedged flow
Date Tue, 14 Jun 2016 18:41:27 GMT

    [ https://issues.apache.org/jira/browse/NIFI-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330130#comment-15330130
] 

Mark Payne commented on NIFI-2015:
----------------------------------

So if there is a FlowFile in the input queue of RouteOnAttribute, then the behavior will be
as follows:
1. Pull FlowFile from queue
2. Determine which relationship to route the FlowFile to and call ProcessSession.transfer
to transfer it
3. Commit the session. Here, we will attempt to generate a Provenance ROUTE event. This will
result in the above stack trace, and since it's an Exception that is thrown in Session commit,
it will automatically roll back the session. This means that the FlowFile is then re-queued.
4. Go back to 1.

Now, I think that the above sequence of steps is correct, as it means that the corrupt FlowFile
will stay in the flow for triage / evaluation by users.
Where I think the problem lives is that if there is an Exception that occurs on Session Commit,
we rollback the session passing the 'penalize' flag as false. (See StandardProcessSession.commit()
where it catches Exception and calls rollback(), around line 432 or so). We should
probably update this to pass true for the 'penalize' flag. If we do that, then the FlowFiles
that were processed will be rolled back, which means that the Processor is free to make progress
on other FlowFiles, while allowing the FlowFile to stay in the flow, in a penalized state.

Thoughts?



> Corrupted flow file leads to a wedged flow
> ------------------------------------------
>
>                 Key: NIFI-2015
>                 URL: https://issues.apache.org/jira/browse/NIFI-2015
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 0.6.1
>            Reporter: Christopher McDermott
>            Assignee: Oleg Zhurakousky
>            Priority: Critical
>             Fix For: 1.0.0, 0.7.0
>
>
> Encountered the following error.  Once encountered the processor in question cannot make
any progress.  The incoming connection cannot be emptied or even listed.  Note this issue
is not to address the corrupted flow file but rather the handling of the corrupted file.
> 2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] o.a.n.p.attributes.UpdateAttribute
> java.lang.IllegalStateException: Cannot create Provenance Event Record because FlowFile
UUID is not set
>         at org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700)
~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>         at org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721)
~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>         at org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412)
~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>         at org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634)
~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>         at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295)
~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>         at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283)
~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>         at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>         at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>         at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>         at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>         at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_45]
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_45]
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message