activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: Need help investigating committed messages resurfacing after restart.
Date Fri, 27 Oct 2017 13:04:08 GMT
Bumping the logging level is key to this; even if you succeed in finding
data files that demonstrate what happened, without the more detailed
logging it may be impossible to tell *why* it happened. If you get a large
enough disk attached to the machine to hold months of logs, that's great,
just make sure that you turn off Log4J's feature to only keep a limited
number of files (I think it's set to 10 by default). Or you could run a
script in a tight loop that looks to see if a log file has rolled over and
copies the newly-rolled-over file to a network file located or to cloud
storage such as S3.

I'd suggest you keep at least the last two sets of saved data files (that
is, don't delete your current set when you analyze it and it doesn't have
the problem, wait till the next one checks out before you delete it),
because once you find the first bad set, you'll want to compare it to the
last good set to see what changed. There's a good chance that you already
thought of this (so please ignore the suggestion if I'm saying something
you already figured out), but since this takes months I'd hate to see you
get to the end and then realize that you don't have everything you
want/need.

I'd also suggest you run the analysis more frequently (hourly, maybe, or
even more often if possible), to try to limit the size of the delta between
the last good and first bad snapshots. The closer they are, the better the
odds that you'll be able to identify which difference between the files is
the relevant difference (because most of the differences will be valid and
unrelated). You're looking for a needle in a haystack and there's a strong
possibility that the needle is invisible and its presence can only be
deduced from context, so anything you can do to shrink the size of the
haystack is to your best interest, especially since this takes so long to
manifest itself.

Sorry I don't have any better ideas for how to analyze this.

Tim

On Oct 27, 2017 1:10 AM, "hakanj" <hakan.johansson@jeppesen.com> wrote:

> It is possible that there is a bug in the ack compaction feature, but if so
> it only shows itself in
> some very special cases. I have tried stressing it, but to no avail. In all
> my tests the broker
> does the right thing.
>
> There are no cron jobs deleting any files in this system. The broker thinks
> that the KahaDB
> database is structurally sound. No complaints about any missing files
> during
> startup.
>
> When we looked in the logs of our other applications we could see that the
> resurrected messages
> were processed as expected, when expected, including a transactional commit
> to the broker. Before
> the broker was taken down the queue was empty, with matching EnqueueCount
> and DequeueCount.
>
> The TRACE level idea is great and will probably show what is happening, but
> I am afraid of running
> out of disk space. The logs would be huge as we need to run the system for
> several months before
> the issue happens. The broker is running on a virtual machine with limited
> available disk. There is
> also the performance impact to take into consideration. We would need to do
> some performance
> measurements before trying this in production. If the performance impact is
> not too bad, then we
> could ask for a proper disk to be added to the machine and move the logging
> there with the TRACE
> enabled.
>
> My current plan is to take a copy of the kahadb directory once a week and
> take a look at it. I have
> found some utilities that shows the contents of those files. Right now the
> error has not yet
> occurred, but it has only been running for a week so far since the last
> restart.
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message