activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clebert Suconic <clebert.suco...@gmail.com>
Subject Re: Artemis use of PhantomReference impacting GC performance
Date Fri, 05 Feb 2016 15:31:58 GMT
... and I did more tests since my last email.. and I can't find
anywhere on our code where we use UnpooledByteBuffer through that
path. So, this is perhaps through the HTTP upgrade.


So, I will need some information from you on how you're creating the
connections. (I would suggest you using direct connections anyway,
since you need high performance and low latency).


Once you give me some update I will resume my investigation.



And thanks for being so through on this thread.. it helped a lot.

On Fri, Feb 5, 2016 at 9:40 AM, Clebert Suconic
<clebert.suconic@gmail.com> wrote:
> I'm a bit confused, because I'm debugging this on master, and at the
> version you are using,
>
>
> And the Allocator there would been PartialPooledByteBufAllocator,
> which will use POOLED on all DirectBuffers. I even ran a couple of
> tests that will send as Netty and the buffer there was always the
> pooled one.
>
>
> Which port are you using for this? Wildfly has one thing that will
> delegate through the HTTP server, upgrade the socket and defer it to
> Artemis as an embedded component. Perhaps there's an issue on that
> integration? on which I'm debugging to find where this is breaking.
>
>
>
>
> On Fri, Feb 5, 2016 at 7:51 AM, jim_b_o <james@inaseq.com> wrote:
>> Thanks to you both for your suggestions and questions.
>>
>> Answers to questions
>>
>>
>> Regarding VM parameters and GC load, I'm confident I have this under
>> control.  I have a lot of experience with GC tuning.  I've been monitoring
>> this specific behaviour for a couple of months now in preparation for moving
>> WildFly into production.  Clebert - you may recall I'm the one who spotted
>> the memory leak in ARTEMIS-320 by observing changed GC behaviour and went to
>> the effort to identify it down to the exact line of code.  So you're right
>> to enquire about this but I know this is not a problem with GC settings.
>>
>> Regarding my testing, I do burn-in for all tests and the JBoss versus
>> WildFly comparisons are on identical hardware using the same JDK version, GC
>> settings and load pattern (and as noted earlier are really only exercising
>> the JMS portions of the container).  I'm not trying to argue that JBoss
>> messaging is better than Artemis but I do believe there is currently an
>> issue of some sort that is worth investigating.
>>
>> Regarding my specific use case and the possibility of making a small
>> reproducer I can provide that if you confirm you still need it after reading
>> the next section.
>>
>> Progress
>>
>>
>> As promised I ran some tests again whilst isolating the JMS and EJB clients.
>> This confirmed for certain that the issue is in the JMS/Netty not the
>> EJB/XNIO.
>>
>> So the high number of PhantomReferences are being allocated in this stack:
>>
>> Thread [Thread-1
>> (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$2@788d198e-832298282)]
>> (Suspended (breakpoint at line 80 in PhantomReference))
>>         Cleaner(PhantomReference<T>).<init>(T, ReferenceQueue<? super
T>)
>> line: 80
>>         Cleaner.<init>(Object, Runnable) line: 115
>>         Cleaner.create(Object, Runnable) line: 133
>>         DirectByteBuffer.<init>(int) line: 139
>>         ByteBuffer.allocateDirect(int) line: 311
>>         UnpooledUnsafeDirectByteBuf.allocateDirect(int) line: 108
>>         UnpooledUnsafeDirectByteBuf.<init>(ByteBufAllocator, int, int) line:
>> 69
>>         UnpooledByteBufAllocator.newDirectBuffer(int, int) line: 50
>>         UnpooledByteBufAllocator(AbstractByteBufAllocator).directBuffer(int,
>> int) line: 155
>>         UnpooledByteBufAllocator(AbstractByteBufAllocator).directBuffer(int)
>> line: 146
>>         NettyServerConnection.createTransportBuffer(int) line: 38
>>
>> RemotingConnectionImpl(AbstractRemotingConnection).createTransportBuffer(int)
>> line: 162
>>         SessionReceiveMessage.encode(RemotingConnection) line: 59
>>         ChannelImpl.send(Packet, boolean, boolean) line: 225
>>         ChannelImpl.sendBatched(Packet) line: 205
>>         CoreSessionCallback.sendMessage(ServerMessage, ServerConsumer, int)
>> line: 84
>>         ServerConsumerImpl.deliverStandardMessage(MessageReference,
>> ServerMessage) line: 883
>>         ServerConsumerImpl.proceedDeliver(MessageReference) line: 366
>>         QueueImpl.proceedDeliver(Consumer, MessageReference) line: 2358
>>         QueueImpl.deliver() line: 1873
>>         QueueImpl.access$1400(QueueImpl) line: 97
>>         QueueImpl$DeliverRunner.run() line: 2581
>>         OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run() line: 100
>>         ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1142
>>         ThreadPoolExecutor$Worker.run() line: 617
>>         Thread.run() line: 745
>>
>> Every java.nio.DirectByteBuffer that is allocated is creating a
>> PhantomReference.
>>
>> Clebert - In your earlier responses you refer to "netty for instance would
>> still be bootstrapping the reusable pool area" and "This could be Netty
>> bootstrapping the PooledByteBuffer arena", however the stack above suggests
>> that these ByteBuffers are not being pooled.  *Note the
>> UnpooledByteBufAllocator*.  This appears to be the default case for Netty
>> 4.0 (http://netty.io/wiki/new-and-noteworthy-in-4.0.html#wiki-h2-3) but is
>> changed to default to pooled in 4.1
>> (http://netty.io/wiki/new-and-noteworthy-in-4.1.html#wiki-h3-7).  So I think
>> this is our underlying problem.  DirectByteBuffers are only recommended for
>> long-lived scenarios (per java.nio.ByteBuffer JavaDocs) which is not the
>> case if they are being used for only one message then discarded.  It looks
>> like the NettyServerConnection should either be using HeapByteBuffers or we
>> should upgrade to Netty 4.1 so that the DirectByteBuffers get pooled and
>> reused.
>>
>> Is there a way to set the Netty ALLOCATOR ChannelOption to
>> PooledByteBufAllocator via the command line so that I could test it with the
>> existing 4.0?
>>
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/Artemis-use-of-PhantomReference-impacting-GC-performance-tp4706961p4707054.html
>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
>
>
> --
> Clebert Suconic



-- 
Clebert Suconic

Mime
View raw message