nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "tara.arent@spglobal.com" <tara.ar...@spglobal.com>
Subject Deployment Strategy Choices - Live flow deployment vs Stop and replace entire flow
Date Thu, 05 Mar 2020 17:28:05 GMT

We have come across an issue around updating existing flows that are currently running in
production.


  1.  We can stop the flows and deploy the entire new flow with all of the custom processors
and all other changes.
     *   Issues

                                                               i.      Data can be lost during
the time of deployment

                                                             ii.      How will this affect
the provenance data – can we reattach to existing data with no issues, do we need to keep
the flow and content data also and reattach that data to the newly installed instances

                                                           iii.      For a cluster how will
this affect the zookeeper if at all

                                                           iv.      State between processors,
will removing the state have any unforeseen issues

     *   Solutions for Streaming Flows & some batch flows

                                                               i.      Add a buffer or Queue
using Kafka or some other queue that will allow stopping of the data into Nifi while we make
the changes to the live flow – works as if we had a connection at the start of every flow

  1.  We can do a live flow deployment using the nifi registry and api and cli, by upgrading
each modified part of the flow piece by piece, we can stop the queue before the first change
and make changes and restart the flow, this will avoid the loss of data that we will see with
the first method
     *   Issues

                                                               i.      Custom controller service
– nypiapi has some tools around starting and stopping the controllers and services will
this address the service name issues when changing a controller

                                                             ii.      This can get very complicates
when the flow has multiple changes that are needed

                                                           iii.      There will be much more
places where we can have a failure in this automated system

                                                           iv.      Are there any flow changes
that we cannot accomplish on a live flow?

     *   Pros

                                                               i.      The validation that
occurs when using the registry to deploy into a live flow will stop the deployment if this
is a breaking change, hopefully that would help avoid issues with the deployments

I was hoping that there are some best practices around updating flows that are in production,
or what has worked for others and what did not.


________________________________

The information contained in this message is intended only for the recipient, and may be a
confidential attorney-client communication or may otherwise be privileged and confidential
and protected from disclosure. If the reader of this message is not the intended recipient,
or an employee or agent responsible for delivering this message to the intended recipient,
please be aware that any dissemination or copying of this communication is strictly prohibited.
If you have received this communication in error, please immediately notify us by replying
to the message and deleting it from your computer. S&P Global Inc. reserves the right,
subject to applicable local law, to monitor, review and process the content of any electronic
message or information sent to or from S&P Global Inc. e-mail addresses without informing
the sender or recipient of the message. By sending electronic message or information to S&P
Global Inc. e-mail addresses you, as the sender, are consenting to S&P Global Inc. processing
any of your personal data therein.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message