activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: What do you think about my LevelDB /Zookeeper Setup?
Date Wed, 04 Mar 2015 13:47:54 GMT
I can't comment on the LevelDB/Zookeeper aspects of the config, but here
are a couple other observations:

4 million messages * 100KB per message is way more than your 10GB
storeUsage limit you've configured.

Since your messages are large and you're concerned about running out of
heap, I'd consider configuring the cursors for your destinations to hold a
relatively small number of messages.  That should ensure that even if you
have lots of messages backed up on disk, they won't use too much heap
space.  The actual size you should use depends on how many destinations you
have.  I'd strongly recommend you test this by storing your theoretical max
of 4 million messages and confirm that the JVM doesn't fall over.

Incidentally, if you really did back up 4 million messages, it's going to
take you 10 days to process them all at 10 messages a second.  Are you sure
both those requirements are right?

Tim

On Mar 3, 2015 12:43 PM, "Marc Schöchlin" <ms@256bit.org> wrote:
>
> Hello ActiveMQ users,
>
> i want to use ActiveMQ;
>
>   * with activemq 5.11.1
>   * with LevelDB/Zookeeper
>   * on three pyhsical nodes
>   * for receiving and sending ~10 Messages per second
>   * a maximum number of 4 million messages (100 KB per message)
>   * with 1GB maxheap
>   * with Java 1.7.075
>
>
> I attached my configuration!
>
> Are there some users which use a production scenario like this?
> Do you recommend this?
> Do you have suggestions for the configuration?
> How can i effectively prevent heap memory problems (poor performance is
> o.k.)?
>
> My configuration looks like this:
>
> activemq.xml
> ------------------------------------------------------------------------
> <?xml version="1.0" encoding="UTF-8"?><beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
>     <!-- Allows us to use system properties as variables in this
> configuration file -->
>     <bean
>
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>         <property name="locations">
>             <value>file:${activemq.conf}/credentials.properties</value>
>         </property>
>     </bean>
>
>     <!--
>         The <broker> element is used to configure the ActiveMQ broker.
>     -->
>     <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="brn" dataDirectory="${activemq.data}">
>
>         <destinationPolicy>
>             <policyMap>
>               <policyEntries>
>                 <!-- QUEUE -->
>                 <policyEntry queue=">" producerFlowControl="true"
> memoryLimit="100mb" maxPageSize="200" maxBrowsePageSize="100"
> lazyDispatch="true">
>                   <deadLetterStrategy>
>                         <individualDeadLetterStrategy queueSuffix=".DLQ"
> useQueueForQueueMessages="true" />
>                   </deadLetterStrategy>
>                   <pendingMessageLimitStrategy>
>                         <constantPendingMessageLimitStrategy limit="100"/>
>                   </pendingMessageLimitStrategy>
>                </policyEntry>
>                <!-- TOPIC -->
>                <policyEntry topic = ">" producerFlowControl = "true"
> memoryLimit = "64mb">
>                   <deadLetterStrategy>
>                     <individualDeadLetterStrategy  topicSuffix=".DLQ" />
>                   </deadLetterStrategy>
>                   <pendingSubscriberPolicy>
>                      <vmCursor/>
>                   </pendingSubscriberPolicy>
>                </policyEntry>
>               </policyEntries>
>             </policyMap>
>         </destinationPolicy>
>
>
>         <!--
>             The managementContext is used to configure how ActiveMQ is
> exposed in
>             JMX. By default, ActiveMQ uses the MBean server that is
> started by
>             the JVM. For more information, see:
>
>             http://activemq.apache.org/jmx.html
>         -->
>       <managementContext>
>             <managementContext createConnector="true"/>
>       </managementContext>
>
>       <persistenceAdapter>
>                 <replicatedLevelDB
>                     directory="/data/apache-activemq/activemq-data/"
>                     replicas="3"
>                     weight="20"
>                     bind="tcp://0.0.0.0:0"
>
> zkAddress="10.228.54.20:2181,10.228.54.24:2181,10.228.54.25:2181"
>                     zkPath="/active-mq/leveldb-stores"
>             hostname="10.228.54.20"
>             sync="local_disk"
>                     indexCompression="snappy"
>             logCompression="snappy"
>       />
>       </persistenceAdapter>
>
>           <!--
>             The systemUsage controls the maximum amount of space the
> broker will
>             use before disabling caching and/or slowing down producers.
> For more information, see:
>             http://activemq.apache.org/producer-flow-control.html
>           -->
>           <systemUsage>
>             <systemUsage>
>                 <memoryUsage>
>                     <memoryUsage percentOfJvmHeap="70"/>
>                 </memoryUsage>
>                 <storeUsage>
>                     <storeUsage limit="10 gb"/>
>                 </storeUsage>
>                 <tempUsage>
>                     <tempUsage limit="1 gb"/>
>                 </tempUsage>
>             </systemUsage>
>         </systemUsage>
>
>         <!--
>             The transport connectors expose ActiveMQ over a given
> protocol to
>             clients and other brokers. For more information, see:
>
>             http://activemq.apache.org/configuring-transports.html
>         -->
>         <transportConnectors>
>             <!-- DOS protection, limit concurrent connections to 1000
> and frame size to 100MB -->
>             <transportConnector name="openwire"
> uri="nio://
0.0.0.0:61616?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
>             <transportConnector name="amqp"
> uri="amqp://
0.0.0.0:5672?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
>             <transportConnector name="stomp"
> uri="stomp://
0.0.0.0:61613?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
>             <transportConnector name="mqtt"
> uri="mqtt://
0.0.0.0:1883?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
>             <transportConnector name="ws"
> uri="ws://
0.0.0.0:61614?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
>         </transportConnectors>
>
>         <!-- destroy the spring context on shutdown to stop jetty -->
>         <shutdownHooks>
>             <bean xmlns="http://www.springframework.org/schema/beans"
> class="org.apache.activemq.hooks.SpringContextHook"/>
>         </shutdownHooks>
>
>     </broker>
>
>     <!--
>         Enable web consoles, REST and Ajax APIs and demos
>         The web consoles requires by default login, you can disable this
> in the jetty.xml file
>
>         Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
>     -->
>     <import resource="jetty.xml"/>
>
> </beans>
> ------------------------------------------------------------------------
>
> zoo.cfg
> ------------------------------------------------------------------------
>
> # The number of milliseconds of each tick
> tickTime=2000
> # The number of ticks that the initial
> # synchronization phase can take
> initLimit=10
> # The number of ticks that can pass between
> # sending a request and getting an acknowledgement
> syncLimit=5
> # the directory where the snapshot is stored.
> # do not use /tmp for storage, /tmp here is just
> # example sakes.
> dataDir=/data/zookeeper/data
> # the port at which the clients will connect
> clientPort=2181
>
> server.1=lnxp-4571.srv.tacatucafoobar.net:2888:3888
> server.2=lnxp-4575.srv.tacatucafoobar.net:2888:3888
> server.3=lnxp-4576.srv.tacatucafoobar.net:2888:3888
>
>
> Regards
> Marc
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message