activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clebert Suconic <clebert.suco...@gmail.com>
Subject Re: [DISCUSS] OutOfDirectMemory on high memory pressure 4 NettyConnection
Date Mon, 06 Mar 2017 12:39:18 GMT
On Mon, Mar 6, 2017 at 4:10 AM, nigro_franz <nigro.fra@gmail.com> wrote:
> Hi guys!
>
> I'm writing here to have your opinion about what could look like an "issue"
> that I've found in the NettyConnection (InVM suffer of the same issue, but
> cover different use cases) of Artemis.
> If you push enough pressure on the NettyConnection::createTransportBuffer
> with huge messages and/or an high count of living connections there will be
> a moment in which Artemis will throw an OutOfDirectMemory exception due to
> the reached limit imposed by the io.netty.maxDirectMemory property.
> I've noticed that the most there will be the speed difference between the
> journalling phase and the messages arrival (through the connection) the most
> it will be the case that the exception will be thrown, but there are
> multiple scenarios that could make it happens.
>
> IMHO this issue is not something that could be solved switching to an
> unpooled Heap ByteBuf allocator: it will shift the responsability to the
> JVM's GC, leading to even worst case effects, like infinite and
> unpredictable GC pauses (eg due to faster aging of garbage and/or homougous
> space exaustions depending by the rate/size/fragmentation of the byte[]
> allocations).
>
> What i'm seeing is more a design needs: an effective way to backpressure the
> different process layers of the messages.
> Netty doesn't provide it if not in the form of that exception, but for an
> application like Artemis I think it need to be done something that can be
> queried, monitored and tuned.
>
> Having an effective way to monitor and bound the latency between each stage
> will enable the lower stages (ie in front of the I/O) to choose a proper
> strategy, like dropping new connections when overloaded or blocking them
> until proper timeouts/freeing of resources.


I think we should use some sort of pool on messages. Reusing the big
pools before the messages are released.

One of the changes as part of my big PR was to not duplicate buffers
on every deliver.. that alone should release some pressure on GC for
cases where the message is delivered to more than one destination
(topic subscriptions).

That should also make it possible to optimize countings.



>
> I've noticed that all the concurrent stages of message processing are backed
> by unbounded queues (ie LinkedBlockingQueue or ConcurrentLinkedQueue) hidden
> by the Executors/ExecutorServices and that's the point that need to be
> improved: switching to bounded queues and real message passing (not
> Runnable) between the stages will let the application to know in a simple
> and effective way when the next stages can't keepup with the processing
> requests.

what would you change here? what would you use for such things?

Mime
View raw message