activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christopher L. Shannon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-4495) Improve cursor memory management
Date Tue, 01 Mar 2016 13:26:18 GMT

    [ https://issues.apache.org/jira/browse/AMQ-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15173744#comment-15173744
] 

Christopher L. Shannon commented on AMQ-4495:
---------------------------------------------

[~gtully],

Two questions/comments,

First, with this new commit, I assume that means there will be reduced performance again?
Some of the brokers I've run are on machines with relatively slow disk performance so that
is a potential concern.

Second, do you think your new change might reduce potential OOM errors?  I've seen out of
memory problems occasionally even though proper usage limits are set and I've always thought
that maybe that had something to do with the fact that paging in on dispatch could load more
than 100% of the usage into memory.

> Improve cursor memory management
> --------------------------------
>
>                 Key: AMQ-4495
>                 URL: https://issues.apache.org/jira/browse/AMQ-4495
>             Project: ActiveMQ
>          Issue Type: Bug
>            Reporter: Dejan Bosanac
>            Assignee: Gary Tully
>             Fix For: 5.9.0
>
>         Attachments: FasterDispatchTest.java
>
>
> As currently stands, the store queue cursor will cache producer messages until it gets
to the 70% (high watermark) of its usage. After that caching stops and messages goes only
in store. When consumers comes, messages get dispatched to it, but memory isn't released until
they are acked. The problem is with the use case where producer flow control is off and we
have a prefetch large enough to get all our messages from the cache. Then, basically the cursor
gets empty and as message acks release memory one by one, we go to the store and try to batch
one message at the time. You can guess that things start to be really slow at that point.

> The solution for this scenario is to wait with batching until we have more space so that
store access is optimized. We can do this by adding a new limit (smaller then the high watermark)
which will be used as the limit after which we start filling cursor from the store again.
> All this led us to the following questions:
> 1. Why do we use 70% as the limit (instead of 100%) when we stop caching producer messages?
> 2. Would a solution that stop caching producer messages at 100% of usage and then start
batching messages from the store when usage drops below high watermark value be enough. Of
course, high watermark would be configurable, but 100% by default so we don't alter any behavior
for regular use cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message