nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aldrin Piri <aldrinp...@gmail.com>
Subject Re: Upgrading NiFi Registry
Date Thu, 28 Jun 2018 19:11:06 GMT
I don't know that it necessarily needs to be changed but the convention we
have in other images where we make use of a wrapper init script gives a
little more flexibility for investigating/manipulating instance internals.
That convention would have made things a bit more straightforward for this
scenario.  In terms of the signals handling, I was a bit surprised they
weren't being picked up by Java but would need to investigate a bit more to
see if it was just my hastiness or if there is something else at play.

On Thu, Jun 28, 2018 at 2:33 PM Kevin Doran <kdoran@apache.org> wrote:

> Thanks Aldrin! That sequence makes sense to me and should result in a
> clean copy of the H2 Database. Mike, you're probably aware of this, but you
> will also want to grab the files in the HOME/flow_storage directory
> assuming those were left I nthe default settings and are not on your Docker
> volume.
>
> Aldrin, is there anything you see regarding the Registry Docker image that
> could be improved, for which we could open a JIRA? Just curious to get your
> insight while it is fresh in your mind.
>
> Thanks!
> Kevin
>
> On 6/28/18, 14:24, "Aldrin Piri" <aldrinpiri@gmail.com> wrote:
>
>     Hi Mike,
>
>     Registry's docker image is a little off from some of the other NiFi
> project
>     configurations but the following approach should be fine and is fairly
>     conservative in how it is carried out.
>
>        1. Exec into the container - *docker exec -it <registry container
>        name/id> /bin/bash*
>        2. Inside container: stop Registry, and have your interactive
> session
>        terminated as PID 1 was terminated - *./bin/nifi-registry.sh stop*
>        3. On your host, copy the h2 file to your host - *docker cp
> <registry
>        container name/id>
>     **/opt/nifi-registry/nifi-registry-0.1.0/database/nifi-registry.mv.db
>        <path on host>*
>        4. Copy the database file in question to your volume for your new
>        instance
>        5. Start the new container to verify import/migration
>        6. Cleanup the old container and files as appropriate
>
>     1 & 2 are questionable in terms of their necessity but with the
> potential
>     for the container to get sigkilled, lend to a safer approach.  The Java
>     process doesn't seem to handle the sigterm from my quick testing of
> docker
>     stop.
>
>     You have the ability to restart where you left off on the original
>     container via *docker start **<registry container name/id>*.
>
>     I did a quick run through of these steps with vanilla versions of
> 0.1.0 and
>     0.2.0 and it seemed to work as anticipated.
>
>     Let us know how it goes.
>
>     On Thu, Jun 28, 2018 at 1:52 PM Kevin Doran <kdoran@apache.org> wrote:
>
>     > Oh right. My mistake. Not sure if there is a way to stop the nifi
> registry
>     > process without stopping the container. Other's more knowledgeable of
>     > Docker might know... In that case I'd say it's worth a shot as that
> DB file
>     > should be valid so long as nothing is being written to it.
>     >
>     > The adoption of the git persistence provider stuff is probably still
> small
>     > given how new it is, but anecdotally, I've been using it for about a
> week
>     > now, syncing to a github repo as a remote, and I've not run into any
> issues
>     > with it nor have I heard of anyone else having a problem with it. It
> seems
>     > pretty solid to me. One of the nice things about the git option is
> that the
>     > format saved to git in the snapshots is compatible with a tool like
> NiFi
>     > CLI, so (while some custom scripting still required), you should be
> in a
>     > pretty good position to do disaster recovery to recreate a NiFi
> Registry
>     > instance with your bucket/flow/flow-snapshot data just from the
> contents of
>     > your git repository should you need to for whatever reason (again,
> would
>     > need to script the right NiFi CLI commands to crawl the git
> directory tree,
>     > but it's pretty straightforward as it's just a matter of parsing
> json and
>     > yml files). One really nice feature is that deletes (a flow or
> bucket for
>     > instance) happen as a new commit so you can even recover objects
> that were
>     > deleted from NiFi Registry by accident, for instance.
>     >
>     > If your team/users are using NiFi Registry and have feedback,
> positive or
>     > otherwise, let us know as it would be great to factor that in to the
>     > existing backlog for what to target in future releases.
>     >
>     > Best,
>     > Kevin
>     >
>     > If your team runs into any trouble with this, do let us know as I'm
> not
>     > sure the deployment footprint
>     >
>     > On 6/28/18, 13:31, "Mike Thomsen" <mikerthomsen@gmail.com> wrote:
>     >
>     >     Kevin,
>     >
>     >     The problem with our situation is that it's all inside Docker so
> I
>     > don't
>     >     think I can just stop the registry and proceed. That's why I was
>     > asking if
>     >     it sounded feasible to just say "hands off for 5 minutes while
> we copy
>     > the
>     >     data."
>     >
>     >     Also, what is the maturity level of the GitFlow option at this
> point?
>     > That
>     >     was particularly attractive as a storage solution because it
> should
>     > make it
>     >     much easier to export everything from our dev team to our end
> users.
>     >
>     >     Thanks,
>     >
>     >     Mike
>     >
>     >     On Thu, Jun 28, 2018 at 1:28 PM Kevin Doran <kdoran@apache.org>
> wrote:
>     >
>     >     > Mike,
>     >     >
>     >     > I don't know the extent of H2's capabilities, but for NiFi
> Registry
>     >     > changing the location of those files will require restarting
> the
>     > server to
>     >     > pick up the new database location from
> nifi-registry.properties,
>     > correct?
>     >     > So if I follow correctly what the objective is, my
> recommendation
>     > would be
>     >     > the following steps:
>     >     >
>     >     > 1. Stop NiFi Registry
>     >     > 2. Move/copy the H2 database files to the Docker volume
> location
>     > (should
>     >     > be safe to do when Registry is stopped)
>     >     > 3. Change nifi-registry.properties to point the database config
>     > props to
>     >     > the new location
>     >     > 4. Start NiFi Registry
>     >     >
>     >     > If the goal is actually just to copy a snapshot of the H2
> database
>     > to the
>     >     > docker volume and not re-point NiFi Registry, which you want
> to keep
>     >     > running, I suppose you could do that. It should be fine to
> copy it if
>     >     > nothing is being written, but of course no guarantees and
> YMMV, so
>     > at the
>     >     > least I'd try to verify that it is a validate db file after
> copying
>     > it,
>     >     > using something like [1].
>     >     >
>     >     > [1]
>     >     >
>     >
> https://stackoverflow.com/questions/32727588/how-to-check-h2-database-health-and-corruption
>     >     >
>     >     > Hope this helps,
>     >     > Kevin
>     >     >
>     >     > On 6/28/18, 12:34, "Mike Thomsen" <mikerthomsen@gmail.com>
> wrote:
>     >     >
>     >     >     Kevin/Bryan,
>     >     >
>     >     >     We have a bunch of users that are using Docker and might
> not
>     > have set
>     >     >     things up to point to a docker volume for the H2 database
>     > location. Do
>     >     >     either of you know if H2 will allow you to copy the files
> over
>     > to the
>     >     >     Docker volume while everything's running?
>     >     >
>     >     >     Thanks,
>     >     >
>     >     >     Mike
>     >     >
>     >     >     On Wed, Jun 20, 2018 at 11:58 AM Bryan Bende <
> bbende@gmail.com>
>     > wrote:
>     >     >
>     >     >     > Ah good point Mark... yes the old db properties are only
> need
>     > this
>     >     >     > first time so that it can auto-migrate the old DB to the
> new
>     > one,
>     >     >     > after that you don't need the old properties anymore.
>     >     >     >
>     >     >     > Thanks Kevin!
>     >     >     >
>     >     >     > On Wed, Jun 20, 2018 at 11:57 AM, Kevin Doran <
>     > kdoran@apache.org>
>     >     > wrote:
>     >     >     > > Thanks Mark and Bryan. I will add a NiFi Registry 0.1
> -> 0.2
>     >     > migration
>     >     >     > guide to include these steps as part of updating the
> site with
>     > news
>     >     > of the
>     >     >     > new release.
>     >     >     > >
>     >     >     > > Thanks,
>     >     >     > > Kevin
>     >     >     > >
>     >     >     > > ________________________________
>     >     >     > > From: Mark Bean <mark.o.bean@gmail.com>
>     >     >     > > Sent: Wednesday, June 20, 2018 8:53:39 AM
>     >     >     > > To: dev@nifi.apache.org
>     >     >     > > Subject: Re: Upgrading NiFi Registry
>     >     >     > >
>     >     >     > > Thanks Bryan. There is actually another step not
> explicitly
>     >     > mentioned. At
>     >     >     > > least for 0.1.0 -> 0.2.0, I needed to modify the
>     >     > nifi-registry.properties
>     >     >     > > file as well. The 0.2.0 version has new
> properties/values
>     > not in
>     >     > the
>     >     >     > 0.1.0.
>     >     >     > > And, I had to set the following for the database (using
>     > values from
>     >     >     > 0.1.0).
>     >     >     > > This was only required on the first startup; these
>     > properties can
>     >     > be
>     >     >     > empty
>     >     >     > > values on subsequent startups.
>     >     >     > > nifi.registry.db.directory=
>     >     >     > > nifi.registry.db.url.append=
>     >     >     > >
>     >     >     > > In the future, if using a database location that is
> external
>     > to the
>     >     >     > > installation directory, is nifi.registry.db.url the
> only
>     > property
>     >     > that
>     >     >     > > needs to be modified?
>     >     >     > >
>     >     >     > >
>     >     >     > > On Wed, Jun 20, 2018 at 11:18 AM Bryan Bende <
>     > bbende@gmail.com>
>     >     > wrote:
>     >     >     > >
>     >     >     > >> Mark,
>     >     >     > >>
>     >     >     > >> The database directory and flow storage directory
are
> where
>     > all
>     >     > the
>     >     >     > >> data are. By default these are created in the root
of
> NiFi
>     >     > Registry,
>     >     >     > >> so depending how you want to set it up you could
move
> those
>     >     >     > >> directories to the new install, or you could set
them
> up to
>     > be
>     >     >     > >> external locations so you don't have to move them
> every
>     > time, or
>     >     > you
>     >     >     > >> could upgrade the lib directory of your current
> install and
>     > leave
>     >     >     > >> everything in place.
>     >     >     > >>
>     >     >     > >> The policies are the same as NiFi... stored in
> users.xml and
>     >     >     > >> authorizations.xml the conf directory, depending
how
> you
>     >     > configured
>     >     >     > >> everything. So just copying those two files over
to
> the new
>     >     > install.
>     >     >     > >>
>     >     >     > >> -Bryan
>     >     >     > >>
>     >     >     > >>
>     >     >     > >> On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean <
>     >     > mark.o.bean@gmail.com>
>     >     >     > wrote:
>     >     >     > >> > How does one upgrade the NiFi Registry?
>     >     >     > >> >
>     >     >     > >> > After unpacking the .tar.gz file, how does one
get
> all
>     > the flows
>     >     >     > >> registered
>     >     >     > >> > in a previous version of NiFi Registry into
the
> newly
>     > installed
>     >     >     > version?
>     >     >     > >> > And, how does one ensure all the policies transfer
> as
>     > well?
>     >     >     > >> >
>     >     >     > >> > Thanks,
>     >     >     > >> > Mark
>     >     >     > >>
>     >     >     >
>     >     >
>     >     >
>     >     >
>     >     >
>     >
>     >
>     >
>     >
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message