activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: kahadb index file larger than journal file
Date Thu, 02 Feb 2017 13:27:51 GMT
Chris,

What will be the behavior for anyone who doesn't delete their index after
upgrading? Will the broker continue to function as-is in that scenario, or
is this a required procedure during an upgrade beyond 5.14.3?

Tim

On Feb 2, 2017 5:02 AM, "Christopher Shannon" <
christopher.l.shannon@gmail.com> wrote:

> Sorry for taking a while to get back to you.
>
> Based on your originally message and after analyzing some index files I
> have actually discovered a bug with KahaDB that causes the index file to
> grow too large when things like out of memory errors occur or any unclean
> shutdown.  On recovery the free pages in the index file are not being
> properly tracked which is leading to excessive index file usage.  So this
> is probably the issue that you are seeing.  I will be fixing this in
> version 5.14.4.  Upon upgrade you will need to delete the old index and
> have it be rebuilt and then going forward it should work as intended.  You
> can track the status of the bug here:
> https://issues.apache.org/jira/browse/AMQ-6590
>
> In terms of the in memory cache, only a fixed number of pages are cached.
> The default is 10 thousand pages and each page is 4KB.  So roughly 40 MB of
> memory will be used at most.  If the number of pages in use grows beyond
> that then the cache will swap pages based on the defined eviction
> strategy.  You can increase the number of pages in memory (which would help
> performance at the cost of eating up memory) by tweaking the indexCacheSize
> value of KahaDB.  See http://activemq.apache.org/kahadb.html for more
> info.
>
> On Tue, Jan 24, 2017 at 11:00 PM, Scorpio <ny@xiangjiabao.com> wrote:
>
> > Christopher:
> >
> >
> > Thank you for your explanation!
> >
> > Hope there would be other proper ways to shrink the index file in future
> > ActiveMQ releases.
> >
> > Once we monitored that our broker is killed by system due to "out of
> > memory".
> >
> > I know that meta cache in memory will be synced into index file at
> > checking points. _*Does this mean there is as much memory as  the
> > db.data file being used?*_
> >
> > Even though our broker is idle,  we still find that it uses up much
> > memory. Could they be relevant associated? Thanks
> >
> >
> > Kind regards,
> >
> > Scorpio
> >
> > On 2017/1/24 20:16, christopher.l.shannon [via ActiveMQ] wrote:
> > > The index grows based off the amount of data it tracks in the journal,
> > > but doesn't shrink.  This means if you had a lot of data in your
> > > journal at some point (several gigabytes, etc) then the index would
> > > have grown in size with enough page files to keep track of all of the
> > > data.  This is usually not an issue because the index size generally
> > > stabilizes at some point relative to the work load of the broker.   If
> > > you really want to shrink it the only way right now is to stop the
> > > broker, delete the index and restart so it does a replay and rebuilds
> > > the index but the size would just grow again as messages flow.
> > >
> > > On Mon, Jan 23, 2017 at 8:06 PM, Scorpio <[hidden email]
> > > </user/SendEmail.jtp?type=node&node=4721300&i=0>> wrote:
> > >
> > > > Hi, All,
> > > >
> > > > In my activemq instance, I find that the kahadb index file is larger
> > > than
> > > > the journal file.
> > > >
> > > > As far as I know, /db.data/ is the meta store, index for the journal
> > > file,
> > > > which should means it is much smaller than the journal file.
> > > However, it
> > > > does not look like that. The index file seems to keep growing while
> the
> > > > journal file is rotated. Do I miss something? Thanks.
> > > >
> > > > Log attached:
> > > > root@host:~/run/apache-activemq-5.13.1/data/kahadb# du * -sh
> > > > 16M     db-16.log
> > > > 1.2G    db.data
> > > > 3.2M    db.redo
> > > > 4.0K    lock
> > > > root@host:~/run/apache-activemq-5.13.1/data/kahadb#
> > > >
> > > > Kind regards,
> > > > Scorpio
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > > http://activemq.2283324.n4.nabble.com/kahadb-index-file-
> > larger-than-journal-file-tp4721284.html
> > > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >
> > >
> > > ------------------------------------------------------------
> ------------
> > > If you reply to this email, your message will be added to the
> > > discussion below:
> > > http://activemq.2283324.n4.nabble.com/kahadb-index-file-
> > larger-than-journal-file-tp4721284p4721300.html
> > >
> > > To start a new topic under ActiveMQ - User, email
> > > ml-node+s2283324n2341805h23@n4.nabble.com
> > > To unsubscribe from kahadb index file larger than journal file, click
> > > here
> > > <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=
> > unsubscribe_by_code&node=4721284&code=bnlAeGlhbmdqaWFiYW8uY29tfDQ3Mj
> > EyODR8MTkzNzMzOTQ5NQ==>.
> > > NAML
> > > <http://activemq.2283324.n4.nabble.com/template/
> > NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> > 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> > BasicNamespace-nabble.view.web.template.NabbleNamespace-
> > nabble.view.web.template.NodeNamespace&breadcrumbs=
> > notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> > 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> > >
> >
> >
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> > nabble.com/kahadb-index-file-larger-than-journal-file-
> > tp4721284p4721315.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message