nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: RecordReader and RecordWriter: development work on ruleengine
Date Thu, 05 Oct 2017 20:52:59 GMT
Uwe,

I don't think there is specific documentation on how to write code
using the record readers and writers, but the best example to look at
would be ConvertRecord....

ConvertRecord actually extends from AbstractRecordProcessor:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractRecordProcessor.java

AbstractRecordProcessor does the following:

- Create a stream callback for a flow file
- Create a reader from the input stream of the flow file
- Create a writer for the output stream of the flow file
- Read each record from the reader and pass it to the writer

For each record it delegates to a "process" method to let sub-classes
take action on the record, but ConvertRecord doesn't need to do
anything because the act of taking a record from a reader for format 1
and passing it to writer for format 2, is conversion in itself.

Hope that helps.

-Bryan


On Thu, Oct 5, 2017 at 2:14 PM, Uwe Geercken <uwe.geercken@web.de> wrote:
> Hello,
>
> I would like to pick up development on my processor that uses a ruleengine. It is working
since several month already but I want to convert the processor to use the new RecordReader/RecordWriter
framework. This way I can use any data format, which is really, really cool and will benefit
it's purpose.
>
> What the processor does is to get a record and run it through the ruleengine to evaluate
results based on business rules. The business rules are written in an external web application.
So the benefit is, that the business logic is not embedded in the code (process). You can
change the logic without touvhing the code/process. The ruleengine uses Java reflection to
construct objects and run methods based on the rules and against the data. Using reflection
I can instantiate any Java object, which makes the ruleengine universally usable. Btw: I have
used the ruleengine with Pentaho ETL, Nifi, Kafka and Hadoop - and from Java code of course.
>
> The ruleengine also knows "actions". Actions are fired based on the the result of rules
(group of rules). So this is a way to fire an event based on logic. Again - all outside of
the code. You can manipulate the records of the flowfile this way, e.g.
>
> So I am also interested in the methods that RecordReader provides to get to the data.
I am confident that I can make a really easy implementation that works for any data because
the framework abstracts everything away that is about data format.
>
> Now looking through code of the existing processors helps and is educational but to get
started a documentation would be very helpful. Maybe you can point me to a sample or documentation
which shows the usage of the api for RecordReader/RecordWriter and how I get a hold on the
data/content of the flowfile.
>
> Thanks for your time. Nifi is getting better and better. I am totally convinced that
the ruleengine will benefit many people and help externalize logic, so that a proper division
of responsibilities is possible: between those that write code/create flows and those that
manage a complete set of logic, which is in fact the knowledge of a company. Central logic
and clean code will benefit both quality and agility (time to implement changes).
>
> Greetings,
>
> Uwe

Mime
View raw message