Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of mteira@indra.es designates
 94.126.240.15 as permitted sender)
Message-ID: <4CD7CDE8.3000007@indra.es>
Date: Mon, 8 Nov 2010 11:16:08 +0100
From: Manuel Teira Paz <mteira@indra.es>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES;
 rv:1.9.2.12) Gecko/20101027 Lightning/1.0b2 Thunderbird/3.1.6
MIME-Version: 1.0
To: ActiveMQ Users <users@activemq.apache.org>
Subject: Producer Flow Control unexpected behaviour
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: quoted-printable

Hello all.

Closely related with the recent thread 'Producer Flow Control - what is
AUTHORITATIVE?', I was making some tests trying to understand how
message cursors work, when I found an odd behaviour, pretty easy (I
guess) to reproduce. Nevertheless, since I'm not very confident with my
understanding of Producer Flow Control, I expose the case to your knowledge=
.

I started an ActiveMQ broker, using jdbc persistence, jmx support and
default (implicit) memory limits. Everything else as default.
With jconsole, I can see the memory limit of broker set to 64MB, as is
the limit for any of the queues I create.
I send three persistent message to a given queue (no consumers on it),
each holding a payload of around 16Mb. What I can see through jconsole is:
- A new queue appears under the 'Queue' JMX hierarchy. This new queue
'test0' have the following remarkable attributes:
   - CursorMemoryUsage: 50335131
   - CursorPercentUsage: 75
   - EnqueueCount: 3
   - MemoryLimit: 67108864
   - MemoryPercentUsage: 75
   in broker attributes, we can see:
   - MemoryLimit: 67108864
   - MemoryPercentUsage: 75

Now, sending a new message to 'test0' queue, leave us with the following
attributes:
  - CursorMemoryUsage: 50335131
  - CursorPercentUsage: 75
  - EnqueueCount: 4
  - MemoryLimit: 67108864
  - MemoryPercentUsage: 75

So, it seems that the cursor stopped caching messages from the storage,
since one more message will make the queue (and hence,the broker)
surpass the 64Mb memory limit. ActiveMQ log shows the related line:
[org.apache.activemq.broker.region.cursors.AbstractStoreCursor] test0
disabling cache on size:3, lastCachedIdSeq: 186 current node seqId: 204

Subsequent attempts to send messages to this queue succeeded, and no
change (but in the EnqueueCount) are observed in the queue attributes.

Now, I send the same 16Mb payload message to a different queue, 'test1',
exhausting the broker memory limit. The new queue shows the following
attributes:
   - CursorMemoryUsage: 16778377
   - CursorPercentUsage: 25
   - EnqueueCount: 1
   - MemoryLimit: 67108864
   - MemoryPercentUsage: 25
   - ProducerFlowControl: true

On the broker side, we had:
  - MemoryLimit: 67108864
  - MemoryPercentUsage: 100

 From now on, any attempt to send a message to test0 or test1, blocks
the producer, since the memory limit was reached, note the log:
[org.apache.activemq.broker.region.Queue] Usage Manager Memory Limit
(67108864) reached on queue://test0. Producers will be throttled to the
rate at which messages are removed from this destination to prevent
flooding it. See http://activemq.apache.org/producer-flow-control.html
for more info
for any attempt to send any new message.


What really worries me about this scenario is:

- Producers were throttled due to a particular way of filling the
queues. Have we only used one queue for the test, and the producer would
never get blocked.
-I didn't expect the broker to apply memory limit considerations for
messages that are to be stored permanently in the store. The cursor
memory usage on the test queues led me to think that the cursor is
holding the complete messages (not just references), by means of some
cache mechanism. This mechanism seems to get disabled once the queue
reaches its memory limit, but messages are not dropped from this cache
to allow the broker to continue working for persistent messages once
those cached messages are shifting the memory usage to its limit.

Is this the behaviour to be expected?


Best regards.
--
Manuel

-


Este correo electr=F3nico y, en su caso, cualquier fichero anexo al mismo, =
contiene informaci=F3n de car=E1cter confidencial exclusivamente dirigida a=
 su destinatario o destinatarios. Si no es vd. el destinatario indicado, qu=
eda notificado que la lectura, utilizaci=F3n, divulgaci=F3n y/o copia sin a=
utorizaci=F3n est=E1 prohibida en virtud de la legislaci=F3n vigente. En el=
 caso de haber recibido este correo electr=F3nico por error, se ruega notif=
icar inmediatamente esta circunstancia mediante reenv=EDo a la direcci=F3n =
electr=F3nica del remitente.
Evite imprimir este mensaje si no es estrictamente necesario.

This email and any file attached to it (when applicable) contain(s) confide=
ntial information that is exclusively addressed to its recipient(s). If you=
 are not the indicated recipient, you are informed that reading, using, dis=
seminating and/or copying it without authorisation is forbidden in accordan=
ce with the legislation in effect. If you have received this email by mista=
ke, please immediately notify the sender of the situation by resending it t=
o their email address.
Avoid printing this message if it is not absolutely necessary.