nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph Witt (JIRA)" <>
Subject [jira] [Commented] (NIFI-238) Add processors to write datasets using Kite
Date Tue, 13 Jan 2015 22:24:35 GMT


Joseph Witt commented on NIFI-238:


Ok so this is good stuff to discuss.

Processor Yielding: The purpose of yielding a processor is that there is something environmental
going on like a configuration file that isn't present yet or a connection that isn't available
or something that is generally considered to be transient and by that processor doing nothing
the issue may resolve itself.

Penalization of a flow file:  If the processor gets a flow file that has some condition of
itself and the thing the processor wants to do which is considered transient then it could
penalize the flow file.  This way the processor is free to keep running on other flow files
. An example of where penalizaton is appropriate would be in PutSFTP where when attemtping
to send a flow file with the same name of some file already present on the remote end then
it needs to wait a while to try again and hopefully that conflict will resolve itself.  

Relying on rollback:  Rollback generally exists to protect the framework from what are largely
programming issues (for the most part) in a given processor.  Rollback is the "what we will
do if some error condition occurs but isn't handled by the processor".  But in general these
error conditions are something a processor implementor should deal with (route things to a
specific relationship for the condition, penalize, yield).  There are cases like OutofMemoryErrors
that could bite a processor for no fault of its own and thus rollback is nice there too. 

Routing to failure (or something appropriate for that processor):  In general a processor
encounters a flow file and does whatever that processor is designed to do.  If the flowfile
it gets is not fit for the preconditions of that processor then routing to a relationship
called 'failure' would be reasonable.  The key is for that processor to document which relationships
it has and for what purposes they receive data.  That way someone composing a flow can come
up with cool and creative ways to put things together based on their context and needs.

All things we need to do a great job of describing in a developer guide.


> Add processors to write datasets using Kite
> -------------------------------------------
>                 Key: NIFI-238
>                 URL:
>             Project: Apache NiFi
>          Issue Type: New Feature
>          Components: Extensions
>            Reporter: Ryan Blue
> I think it would be great to have a set of processors that parse incoming flow files
and add the data to Kite datasets.

This message was sent by Atlassian JIRA

View raw message