activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <>
Subject Re: ActiveMQ master-slave topology issue[BUG]
Date Mon, 15 Jun 2015 14:07:19 GMT
It seems pretty clear that the assumption that acquiring a single file lock
without doing any further checks will provide thread-safety in all cases is
not an accurate one.

As I see it, here are the failures of the existing approach:

   - Failure #1 is that when an NFS failure occurs, the master broker never
   recognizes that an NFS failure has occurred and never recognizes that the
   slave broker has replaced it as the master.  The broker needs to identify
   those things even when it has no other reason to write to NFS.
   - Failure #2 is that the slave broker believes that it can immediately
   become the master.  This wouldn't be a problem if the master broker
   instantaneously recognized that it has been supplanted and immediately
   ceded control, but assuming anything happens instantaneously (especially
   across multiple machines) is pretty unrealistic.  This means there will be
   a period of unresponsiveness when a failover occurs.
   - Failure #3 is that once the master recognizes that it no longer is the
   master, it needs to abort all pending writes (in a way that guarantees that
   the data files will not be corrupted if NFS returns when some have been
   cancelled and others have not).

I've got a proposed solution that I think will address all of these
failures, but hopefully others will see ways to improve it. Here's what
I've got in mind:

   1. It is no longer sufficient to hold an NFS lock on the DB lock file.
   In order to maintain master status, you must successfully write to the DB
   lock file within some time period. If you fail to do so within that time
   period, you must close all use of the DB files and relinquish master status
   to another broker.
   2. When the master shuts down during normal NFS circumstances, it will
   delete the DB lock file.  If at any point a slave broker sees that there is
   no DB lock file or that the DB lock file is so stale that the master must
   have shut down (more on that later), it may immediately attempt to create
   one and begin writing to it.  If that write succeeds, it is the master.
   3. All brokers should determine whether the current master is still
   alive by checking the current content of the DB lock file against the
   content read the last time you checked, rather than simply locking the file
   and assuming that tells you who's got ownership. This means that the
   current master needs to update some content in the DB lock file to make it
   unique each on each write; I propose that the content of the file be the
   broker's host, the broker's PID, the current local time on the broker at
   the time it did its write, and a UUID that will guarantee uniqueness of the
   content from write to write even in the face of time issues.  Note that
   only the UUID is actually required for this algorithm to work, but I think
   that having the other information will make it easier to troubleshoot.
   4. Because time can drift between machines, it is not sufficient to
   compare the write date on the DB lock file with your host's current time
   when determining that a file is stale; you must successfully read the file
   repeatedly over a time period and receive the same value each time in order
   to decide that the DB lock file is stale.
   5. The master should use the same approach as the slaves to determine if
   it's still in control, by checking for changes to the content of the DB
   lock file. This means the master needs to positively confirm that each
   periodic write to the DB lock file succeeded by reading it back (in a
   separate thread, using a timeout on the read operation to identify
   situations where NFS doesn't respond), rather than simply assuming that its
   call to write() worked successfully.
   6. When a slave determines that the master has failed to write to the DB
   lock file for longer than the timeout, it attempts to acquire the write
   lock on the DB lock file and write to it to become the master.  If it
   succeeds (because the master has lost NFS connectivity but the slave has
   not), it is the master.  If it fails because another slave acquired the
   lock first or because the slave has lost NFS connectivity, it goes back to
   monitoring for the master to fail to write to the DB lock file.
   7. If the master finds that it has failed to write to the DB lock file
   for some amount of time (which must be less than the overall timeout), it
   shuts down its use of the database files in a way that will avoid
   corrupting the data files no matter how NFS connectivity comes and goes
   during the shutdown process.  The master must not write to the DB lock file
   once it begins shutting down even if NFS connectivity returns during the
   shutdown, because that would indicate to the slaves that it is alive and
   retaining ownership, which is not the case when shutting down. Note: this
   may be difficult or impossible to do, and I don't know either KahaDB or NFS
   at all to know how possible this is or what other options we could use, so
   please give feedback for how best to achieve this goal.
   8. Since we can't rely on the former master having any NFS/network
   connectivity and being able to write any information to indicate that
   shutdown completed, I think there has to be a time limit ("the master
   broker must complete its shutdown within X amount of time, after which the
   slaves can assume they may take ownership of the data files"), which should
   probably be configurable so you can tune that window based on how quickly
   your particular broker, running under your particular load on your
   particular hardware, can shut down.  To facilitate setting that value
   effectively, we should probably log how long shutdown took, so that users
   can test that and set their timeout appropriately.
   9. The system must enforce that the master's inactivity identification
   interval plus the time to shutdown <= the amount of time a slave must see
   the DB lock file be stale before becoming the master.
   10. Once the master has completed shutting down, the slaves can attempt
   to become the master by successfully writing to the DB lock file, which
   will allow them to take ownership of the data files and begin accepting
   connections.  The new master must confirm that it owns the DB lock file
   (either with an explicit NFS lock test, if such things exist, or by
   confirming that the current content of the file is the content that it
   wrote most recently) before accepting connections or beginning to write to
   the data files.

Thoughts?  Suggestions?

One question I still had was whether it was actually realistic to think
that we can always safely cancel the threads of the master broker without
risking corrupting the data files.  Should we make a copy of the data files
(which has performance and storage implications) and have the new master
work from those files instead of the ones the old master will be writing
to, so they can't step on each other?


> Anuj
> Have a look at
> You cannot avoid to have either two or zero master-brokers online during a
> failover. The question is how long this situation lasts (see Arthur
> Naseef's comment on AMQ-5549).
> In my failover-tests with NFS shared storage I was able to reproduce very
> different scenarios:
> - the former master broker *never* shuts down
> - the former master broker shuts down after 15 minutes
> - the former master broker shuts down after 20 seconds
> The only difference between these scenarios were NFS settings. My overall
> impression is that the failover only works with a highly available shared
> storage. As soon as one or multiple brokers lose the NFS connection, the
> situation is getting crazy and I even "managed" it to corrupt the
> persistence store during my tests.
> Also notice the both-brokers-down-problem (
> that I discovered during
> my
> tests.
> Cheers
> Stephan
> On Thu, Apr 30, 2015 at 2:40 PM, Tim Bain <> wrote:
> > An NFS problem was the first thing I thought of when I saw out-of-order
> log
> > lines, especially since you've had that problem before.  And this outage
> > lasted for over two minutes (which doesn't count as "slow" in my book;
> > that's "unavailable" or "down" to me), which is pretty crazy; hopefully
> > your ops team has looked into how that happened and taken steps to ensure
> > it doesn't happen again.
> >
> > A NFS outage does justify a failover to the backup broker; to understand
> > why, think about what prevents failover during normal operation.  The
> > master broker holds a file system lock on a DB lock file, and the slave
> > broker tries repeatedly to acquire the same lock.  As long as it can't,
> it
> > knows the master broker is up and it can't become the master; at the
> point
> > where the lock disappears because the master broker can't access NFS, the
> > slave becomes active (at least, if it can access NFS; if not, then it
> > doesn't know that it could become active and it can't read the messages
> > from disk anyway).  This is exactly what you would want to happen.
> >
> > The real problem here is the one in your last paragraph: when the slave
> > acquires the lock because the master can't access NFS, the master isn't
> > detecting that and becoming the slave.  I'd suggest you try to recreate
> > this failure (in a dev environment) by causing the master broker to be
> > unable to access NFS and confirming that the master remains active even
> > after the slave becomes the master.  Assuming that happens, submit a JIRA
> > bug report to describe the problem.  Make sure you provide lots of
> details
> > about your NFS setup (include version numbers, file system type, etc.)
> and
> > about the O/Ses of the machines the brokers run on, since the behavior
> > might vary based on some of those things and you want to make sure that
> > whoever investigates this can reproduce it.  But make sure you can
> > reproduce it first.
> >
> > Tim
> > Hi,
> >
> > I got the logs in this order only and after further checking the system I
> > got to know that NFS(where we put kahadb and broker logs) was slow during
> > that time.
> >
> > I can understand the delay in logs or I/O operations are slow during that
> > time but it does not justify why failover also open it's transport
> > connector.
> >
> > The main concern here is that the (master-slave-shared-storage)topology
> is
> > broken which should not happen in any case. If I/O operations are not
> > happening, master broker should stop and let the failover serve the
> clients
> > but here master didn't stop and both opened the connector.
> >
> > Thanks,
> > Anuj
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> >
> > Sent from the ActiveMQ - User mailing list archive at
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message