activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clebert Suconic <clebert.suco...@gmail.com>
Subject Re: Artemis use of PhantomReference impacting GC performance
Date Fri, 05 Feb 2016 14:40:39 GMT
I'm a bit confused, because I'm debugging this on master, and at the
version you are using,


And the Allocator there would been PartialPooledByteBufAllocator,
which will use POOLED on all DirectBuffers. I even ran a couple of
tests that will send as Netty and the buffer there was always the
pooled one.


Which port are you using for this? Wildfly has one thing that will
delegate through the HTTP server, upgrade the socket and defer it to
Artemis as an embedded component. Perhaps there's an issue on that
integration? on which I'm debugging to find where this is breaking.




On Fri, Feb 5, 2016 at 7:51 AM, jim_b_o <james@inaseq.com> wrote:
> Thanks to you both for your suggestions and questions.
>
> Answers to questions
>
>
> Regarding VM parameters and GC load, I'm confident I have this under
> control.  I have a lot of experience with GC tuning.  I've been monitoring
> this specific behaviour for a couple of months now in preparation for moving
> WildFly into production.  Clebert - you may recall I'm the one who spotted
> the memory leak in ARTEMIS-320 by observing changed GC behaviour and went to
> the effort to identify it down to the exact line of code.  So you're right
> to enquire about this but I know this is not a problem with GC settings.
>
> Regarding my testing, I do burn-in for all tests and the JBoss versus
> WildFly comparisons are on identical hardware using the same JDK version, GC
> settings and load pattern (and as noted earlier are really only exercising
> the JMS portions of the container).  I'm not trying to argue that JBoss
> messaging is better than Artemis but I do believe there is currently an
> issue of some sort that is worth investigating.
>
> Regarding my specific use case and the possibility of making a small
> reproducer I can provide that if you confirm you still need it after reading
> the next section.
>
> Progress
>
>
> As promised I ran some tests again whilst isolating the JMS and EJB clients.
> This confirmed for certain that the issue is in the JMS/Netty not the
> EJB/XNIO.
>
> So the high number of PhantomReferences are being allocated in this stack:
>
> Thread [Thread-1
> (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$2@788d198e-832298282)]
> (Suspended (breakpoint at line 80 in PhantomReference))
>         Cleaner(PhantomReference<T>).<init>(T, ReferenceQueue<? super
T>)
> line: 80
>         Cleaner.<init>(Object, Runnable) line: 115
>         Cleaner.create(Object, Runnable) line: 133
>         DirectByteBuffer.<init>(int) line: 139
>         ByteBuffer.allocateDirect(int) line: 311
>         UnpooledUnsafeDirectByteBuf.allocateDirect(int) line: 108
>         UnpooledUnsafeDirectByteBuf.<init>(ByteBufAllocator, int, int) line:
> 69
>         UnpooledByteBufAllocator.newDirectBuffer(int, int) line: 50
>         UnpooledByteBufAllocator(AbstractByteBufAllocator).directBuffer(int,
> int) line: 155
>         UnpooledByteBufAllocator(AbstractByteBufAllocator).directBuffer(int)
> line: 146
>         NettyServerConnection.createTransportBuffer(int) line: 38
>
> RemotingConnectionImpl(AbstractRemotingConnection).createTransportBuffer(int)
> line: 162
>         SessionReceiveMessage.encode(RemotingConnection) line: 59
>         ChannelImpl.send(Packet, boolean, boolean) line: 225
>         ChannelImpl.sendBatched(Packet) line: 205
>         CoreSessionCallback.sendMessage(ServerMessage, ServerConsumer, int)
> line: 84
>         ServerConsumerImpl.deliverStandardMessage(MessageReference,
> ServerMessage) line: 883
>         ServerConsumerImpl.proceedDeliver(MessageReference) line: 366
>         QueueImpl.proceedDeliver(Consumer, MessageReference) line: 2358
>         QueueImpl.deliver() line: 1873
>         QueueImpl.access$1400(QueueImpl) line: 97
>         QueueImpl$DeliverRunner.run() line: 2581
>         OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run() line: 100
>         ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1142
>         ThreadPoolExecutor$Worker.run() line: 617
>         Thread.run() line: 745
>
> Every java.nio.DirectByteBuffer that is allocated is creating a
> PhantomReference.
>
> Clebert - In your earlier responses you refer to "netty for instance would
> still be bootstrapping the reusable pool area" and "This could be Netty
> bootstrapping the PooledByteBuffer arena", however the stack above suggests
> that these ByteBuffers are not being pooled.  *Note the
> UnpooledByteBufAllocator*.  This appears to be the default case for Netty
> 4.0 (http://netty.io/wiki/new-and-noteworthy-in-4.0.html#wiki-h2-3) but is
> changed to default to pooled in 4.1
> (http://netty.io/wiki/new-and-noteworthy-in-4.1.html#wiki-h3-7).  So I think
> this is our underlying problem.  DirectByteBuffers are only recommended for
> long-lived scenarios (per java.nio.ByteBuffer JavaDocs) which is not the
> case if they are being used for only one message then discarded.  It looks
> like the NettyServerConnection should either be using HeapByteBuffers or we
> should upgrade to Netty 4.1 so that the DirectByteBuffers get pooled and
> reused.
>
> Is there a way to set the Netty ALLOCATOR ChannelOption to
> PooledByteBufAllocator via the command line so that I could test it with the
> existing 4.0?
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Artemis-use-of-PhantomReference-impacting-GC-performance-tp4706961p4707054.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.



-- 
Clebert Suconic

Mime
View raw message