nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Ryan <brett.r...@gmail.com>
Subject Re: Enrichment plugin for adding attributes from SQL
Date Thu, 04 Jan 2018 21:43:41 GMT
Cool, thanks for the help. I’ll investigate implementing this as a LookupService as there’s
no present service for connecting to SQL, I think; man, I could be wrong, I did try looking
at all available ;)

To help me learn all the processors I actually dragged every processor into process groups
of all related processors just so I could help remember each of the tools.

I like your analogy to unix tools, however; I probably find the tools used to chain tasks
considerably less and conditional expressions easier to work out :)

> On 5 Jan 2018, at 08:32, Andy LoPresto <alopresto@apache.org> wrote:
> 
> UpdateAttribute doesn’t pull from a database, it uses static or dynamic attribute values
and supports NiFi Expression Language. In your original message, you didn’t mention any
database interaction, so I thought you were just trying to accomplish "I wanted to add some
attributes to a FlowFile while not altering the contents of that FlowFile”, which is indeed
what UpdateAttribute does. 
> 
> If you need to retrieve those values from a database, as Joey mentions, the LookupService
is the right tool. 
> 
> With your prior setup, the distributed map cache is as secure as the NiFi configuration
— if using secured NiFi, the communication between that node and any other is over TLS,
and within the node it’s a memory access. 
> 
> A big part of the NiFi philosophy is the same as the Unix philosophy — each tool should
do one job very well, and to perform complicated tasks, chain those tools together. This helps
with provenance reporting, usage reporting, debugging, flow development lifecycle, maintenance,
etc. A processor which retrieves attributes from a database and updates the incoming flowfile
with them is certainly useful in the use case you describe, but is not a generic pattern.
There’s no intent to discourage custom development, and whatever makes your flow work is
fine. Just explaining why you likely won’t see a solution like that in the NiFi bundled
components. 
> 
> 
> Andy LoPresto
> alopresto@apache.org
> alopresto.apache@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Jan 4, 2018, at 4:07 PM, Brett Ryan <brett.ryan@gmail.com> wrote:
>> 
>> Thanks Andy, how would update attribute be able to get the value from sql?
>> 
>> Consider a flow where a piece of information needs to be obtained from a DB but i
do not want the contents of the current FF to be altered, using ExecuteSQL anywhere prior
would not be possible due to replacing the FF contents.
>> 
>> What I had was two seperate flows, one that updates an oauth key in a db keeping
it fresh, the main flow would then read the db just before an invokehttp.
>> 
>> I was originally using a distributed map cache but had concerns that it might not
be secure and was also advised the cache server has been known to go down.
>> 
>>> On 5 Jan 2018, at 04:01, Joey Frazee <joey.frazee@icloud.com> wrote:
>>> 
>>> Andy, Brett,
>>> 
>>> Taking a quick glance at the code it looks like it's enriching attributes from
a database according to a query.
>>> 
>>> If that's correct, there's a LookupAttribute processor that delegates lookups
to a "LookupService" and adds attributes without altering content. There are a variety of
these LookupServices included. I think what you implemented would make sense as a LookupService
and then you could just configure the processor to use that. It could also be used with LookupRecord
then too so there'd be a double payoff.
>>> 
>>> -joey
>>> 
>>>> On Jan 4, 2018, 8:44 AM -0800, Andy LoPresto <alopresto@apache.org>,
wrote:
>>>> Hi Brett,
>>>> 
>>>> It’s great that you found it easy to write a new processor for Apache NiFi.
It is probably an indicator that we need to improve education/evangelism/documentation, however,
that you did not find UpdateAttribute [1], which should do exactly what you were looking for.
>>>> 
>>>> [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/org.apache.nifi.processors.attributes.UpdateAttribute/index.html
>>>> 
>>>> Andy LoPresto
>>>> alopresto@apache.org
>>>> alopresto.apache@gmail.com
>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>> 
>>>>> On Jan 4, 2018, at 7:03 AM, Brett Ryan <brett.ryan@gmail.com> wrote:
>>>>> 
>>>>> Hi all, having used NiFi for a couple days I wanted to add some attributes
to a FlowFile while not altering the contents of that FlowFile.
>>>>> 
>>>>> I had suggestions to use a script processor but that just sounded like
a hack which could become a nuisance to replicate.
>>>>> 
>>>>> Anyway, I figured I'd write a processor to do this, anyone interested
you can find it here.
>>>>> 
>>>>> https://github.com/brettryan/nifi-drunken-bundle
>>>>> 
>>>>> Feel free to do with it what you please.
>>>>> 
>>>>> I've published to maven central but it will take a day to appear in the
search.
>>>>> 
>>>>> <dependency>
>>>>>  <groupId>com.drunkendev</groupId>
>>>>>  <artifactId>nifi-drunken-nar</artifactId>
>>>>>  <version>1.0.0</version>
>>>>>  <type>nar</type>
>>>>> </dependency>
>>>>> <dependency>
>>>>>  <groupId>com.drunkendev</groupId>
>>>>>  <artifactId>nifi-drunken-processors</artifactId>
>>>>>  <version>1.0.0</version>
>>>>> </dependency>
>>>>> <dependency>
>>>>>  <groupId>com.drunkendev</groupId>
>>>>>  <artifactId>nifi-drunken-bundle</artifactId>
>>>>>  <version>1.0.0</version>
>>>>>  <type>pom</type>
>>>>> </dependency>
>>>>> 
>>>> 
> 

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message