activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nigro_franz <nigro....@gmail.com>
Subject Re: [DISCUSS] OutOfDirectMemory on high memory pressure 4 NettyConnection
Date Mon, 06 Mar 2017 15:24:10 GMT
Thanks for the feedback Clebert!

>I think we should use some sort of pool on messages. Reusing the big 
>pools before the messages are released. 
It could work!
Just thinking loud; in order to implement this pool efficiently my advice is
to not use a solution similar to the Netty pool, in which the object
lifecycle could span between threads: Netty itself shows that it doesn't
scale well for high throughput scenarios and risks to be error prone on
leaking messages (it needs to implement a leak detector etc etc).
I've already built some experiments around this approach if anyone is
interested in the results, but could worth try it too!

Instead I'm proposing another form of pooling, that will provide the same
effect of recycling memory but as a consequence and not as a mean: bounded
queues between the message process stages.
Having bounded concurrent queues will provide the pooling for free and a
mean to measure the utilization between all the concurrent stages (eg
incoming,encoding, journaling ,decoding, outgoing etc etc) recognizing when
the system is overloaded in any of its part and letting the dev/user to tune
it consequently.

>One of the changes as part of my big PR was to not duplicate buffers 
>on every deliver.. that alone should release some pressure on GC for 
>cases where the message is delivered to more than one destination 
>(topic subscriptions). 
And it is a super plus to improve the throughput for sure (I've already
benchmarked it, delivering outstanding improvements indeed ): bound the
memory and the max latency is only an additional tool to exploit further
this improved throughput, making it sustainable.

>what would you change here? what would you use for such things? 
I would bound first (putting a bounded queue in front of) the stages that
could be slowed down by external factors (like I/O speeds), that need to
inform the previous ones of being overloaded and that could benefit by any
form a batching:
1) the incoming/outcoming data phase of the connections (InVM or Netty)
2) the management of the journal (the write operations in particular)
makes sense?






--
View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-OutOfDirectMemory-on-high-memory-pressure-4-NettyConnection-tp4722964p4722999.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Mime
View raw message