nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy LoPresto <alopre...@apache.org>
Subject Re: Parameterizing the nifi flow
Date Sat, 12 Aug 2017 00:36:28 GMT
The variable registry is a great tool for parameterizing values that differ between environments/deployments.
In this case it sounds like setting up a flow that read from a flowfile attribute/content
to determine the source table’s name is a better fit. You can create a master list of all
source tables and store it as a plaintext file, then on your schedule, read the contents of
that file, split the content by line, and send each 100+ resulting flowfiles into the part
of the flow that reads from the database. Each would provide the name of the source table
to read at that time.

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Aug 11, 2017, at 3:26 PM, Andrew Grande <aperepel@gmail.com> wrote:
> 
> Hi,
> 
> Read up on the variable registry in the docs, that sounds like a good fit. I don't remember
if it were available in 1.1 though.
> 
> Andrew
> 
> 
> On Fri, Aug 11, 2017, 5:12 PM More, Vikram (CONT) <Vikram.More@capitalone.com <mailto:Vikram.More@capitalone.com>>
wrote:
> Hi,
> 
> I have a nifi flow which pulls/extracts from source database table and loads into target
database table. This flow will run several times in a day to get delta records from source
table (more like batch process running every 3-4 hrs). Now I need to replicate this same process
for 100+ different source tables. So rather than creating 100+ nifi flows for each separate
table, can I create main flow (let's say template) and pass parameter like source extract
sql, target load sql to main flow. And repeat these steps for each source table . Has anyone
tried parameterizing the nifi flows, can you please advice . We are using NiFi 1.1.0
> 
> Appreciate any thoughts here.
> 
> 
> 
> 
> Thanks & Regards,
> 
> Vikram
> 
> 
> 
> 
> The information contained in this e-mail is confidential and/or proprietary to Capital
One and/or its affiliates and may only be used solely in performance of work or services for
Capital One. The information transmitted herewith is intended only for use by the individual
or entity to which it is addressed. If the reader of this message is not the intended recipient,
you are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.


Mime
View raw message