pulsar-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [pulsar] rdhabalia opened a new pull request #6634: [pulsar-broker] add flag to skip broker shutdown on transient OOM
Date Sun, 29 Mar 2020 21:39:09 GMT
rdhabalia opened a new pull request #6634: [pulsar-broker] add flag to skip broker shutdown
on transient OOM
URL: https://github.com/apache/pulsar/pull/6634
 
 
   ### Motivation
   
   Some time due to high dispatch rate on one of the topic can temporarily cause broker to
go OOM and it will be transient error and broker can recover within a few seconds as soon
as some memory gets released. However, 2.4 release has change #4196 which restarts broker
on OOM which can cause huge instability in cluster where that topic moves from one broker
to another and restarts multiple brokers and cause disruption for other topics as well. we
have seen similar kind of issue mentioned at  #5896 . This could be transient error and we
need a way to ignore this error. So, we need a dynamic flag to skip broker shutdown on OOM
to avoid instability in a cluster.
   ```
   01:48:49.549 [pulsar-io-22-37] ERROR org.apache.pulsar.PulsarBrokerStarter - -- Shutting
down - Received OOM exception: Direct buffer memory
   java.lang.OutOfMemoryError: Direct buffer memory
           at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]
           at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[?:?]
           at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
           at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:769) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:745) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.buffer.PoolArena.allocate(PoolArena.java:226) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324)
~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:164)
~[bookkeeper-common-allocator-4.9.4.2-yahoo.jar:4.9.4.2-yahoo]
           at org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:158)
~[bookkeeper-common-allocator-4.9.4.2-yahoo.jar:4.9.4.2-yahoo]
           at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1912) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:1923) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:826) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:797) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:778) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)
~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at org.apache.pulsar.broker.service.Consumer.lambda$sendMessages$51(Consumer.java:265)
~[pulsar-broker-2.4.6-yahoo.jar:2.4.6-yahoo]
           at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
[netty-all-4.1.32.Final.jar:4.1.32.
   :
   : 
   01:48:49.549 [pulsar-io-22-39] ERROR org.apache.pulsar.PulsarBrokerStarter - -- Shutting
down - Received OOM exception: Direct buffer memory
   java.lang.OutOfMemoryError: Direct buffer memory
           at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]
           at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[?:?]
           at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
           at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:769) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:745) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.buffer.PoolArena.allocate(PoolArena.java:226) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324)
~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
~[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:164)
[bookkeeper-common-allocator-4.9.4.2-ya
   hoo.jar:4.9.4.2-yahoo]
           at org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:158)
[bookkeeper-common-allocator-4.9.4.2-yahoo.jar:4.9.4.2-yahoo]
           at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
[netty-all-4.1.32.Final.jar:4.1.32.Final]
           at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
[netty-all-4.1.32.Final.jar:4.1.32.Final]
   ```
   
   ### Modification
   Add dynamic flag to avoid broker shutdown on OOM.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message