nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "matthewmichala@northwesternmutual.com" <matthewmich...@northwesternmutual.com>
Subject NiFi - How to Wait until all Inserts into Table are done, then Create/Analyze Indexes on the Table afterwards
Date Tue, 02 Apr 2019 22:56:21 GMT
NiFi Developers,

I tried reaching out to Stack Overflow Apache-Nifi Tag, but am getting no replies - so normally
I wouldn't ask this email list this type of question, but am not sure where else to turn?

So to keep simple, my 1st Process Group (PG) Truncates then drops indexes in table. Then that
output port routes to input port in 2nd process group the does the 500K Inserts into the table.
After successfully inserting the 500K rows, I want to create the indexes on the table and
analyze it via 3rd process group. This is typical Data Warehouse methodology. Can anyone please
give advice on how to do this?  I'd like to be able to not even start the 3rd process group,
until first 2 complete successfully, and the Insert Count matches the Select Count.

I've tried setting counters, then comparing total inserted count to match initial select count,
but it appears that I cannot reference Counters in the Expression Language because this "${InsertCounter}"
 syntax always returns nulls??   So maybe instead I should be using the Wait & Notify
Processors, so could Wait until all the inserts are done, then Notify the next process (the
create & analyze analyze indexes process) to start after all the inserts are done?  Here's
example code on that from Stack Overflow:

In the wait processor set the Target Signal Count to ${fragment.count}.

Set the Release Signal Identifier in both the notify and wait processor to ${fragment.identifier}
Thanks for any help anyone can provide!

Thanks & Regards,

Matt Michala
Senior Engineer
Digital Workplace & Corp. Solutions
Northwestern Mutual (T07-150)
414-661-4668



This e-mail and any attachments may contain confidential information of Northwestern Mutual.
If you are not the intended recipient of this message, be aware that any disclosure, copying,
distribution or use of this e-mail and any attachments is prohibited. If you have received
this e-mail in error, please notify Northwestern Mutual immediately by returning it to the
sender and delete all copies from your system. Please be advised that communications with
{SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications
that do not have this tag may not be secure and could be observed by a third party. Thank
you for your cooperation. 



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message