camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michele <michele.mazzi...@finconsgroup.com>
Subject Re: Best Strategy to process a large number of rows in File
Date Wed, 30 Mar 2016 14:43:28 GMT
Hi,

yes... it is clear ;)!

I changed configuration of Heap and Perm progressively as you suggested like
this:

-server -Xms1024M -Xmx2048M -XX:PermSize=256M -XX:MaxPermSize=512M  -Xss512M 
-XX:+HeapDumpOnOutOfMemoryError -verbose:gc -Xloggc:gc.log ...

The number of rows processed has increased, unfortunately the result is the
same OOM.

With VisualVM, I have traced Threads and Memory and the biggest object found
is 

org.apache.activemq.broker.region.Queue#1  Retained Size Memory
1.449.034.641 B

I attached images of heap dump of import result for further information

biggest-objects.png
<http://camel.465427.n5.nabble.com/file/n5780047/biggest-objects.png>  
memory-threads.png
<http://camel.465427.n5.nabble.com/file/n5780047/memory-threads.png>  

And then in log of gc, i found this

"ActiveMQ Transport: tcp:///127.0.0.1:63664@61616" daemon prio=5 tid=330
RUNNABLE
	at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48)
	at
org.apache.activemq.util.DataByteArrayOutputStream.ensureEnoughBuffer(DataByteArrayOutputStream.java:249)
	at
org.apache.activemq.util.DataByteArrayOutputStream.writeBoolean(DataByteArrayOutputStream.java:140)
	at
org.apache.activemq.openwire.v6.BaseDataStreamMarshaller.looseMarshalByteSequence(BaseDataStreamMarshaller.java:628)
	   Local Variable: org.apache.activemq.util.ByteSequence#52
	at
org.apache.activemq.openwire.v6.MessageMarshaller.looseMarshal(MessageMarshaller.java:296)
	at
org.apache.activemq.openwire.v6.ActiveMQMessageMarshaller.looseMarshal(ActiveMQMessageMarshaller.java:111)
	at
org.apache.activemq.openwire.v6.ActiveMQObjectMessageMarshaller.looseMarshal(ActiveMQObjectMessageMarshaller.java:111)
	   Local Variable: org.apache.activemq.util.DataByteArrayOutputStream#2
	at
org.apache.activemq.openwire.OpenWireFormat.marshal(OpenWireFormat.java:155)
	   Local Variable:
org.apache.activemq.openwire.v6.ActiveMQObjectMessageMarshaller#1
	   Local Variable: org.apache.activemq.openwire.OpenWireFormat#2
	at
org.apache.activemq.leveldb.DelayableUOW.create_message_record$1(DBManager.scala:299)
	at org.apache.activemq.leveldb.DelayableUOW.enqueue(DBManager.scala:311)
	   Local Variable: org.apache.activemq.command.MessageId#42
	at
org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.doAdd(LevelDBStore.scala:773)
	   Local Variable: java.util.concurrent.atomic.AtomicLong#133
	   Local Variable: java.lang.Long#47
	at
org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore$$anonfun$asyncAddQueueMessage$1.apply(LevelDBStore.scala:785)
	at
org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore$$anonfun$asyncAddQueueMessage$1.apply(LevelDBStore.scala:784)
	at org.apache.activemq.leveldb.LevelDBStore.withUow(LevelDBStore.scala:735)
	   Local Variable: org.apache.activemq.leveldb.DelayableUOW#1
	   Local Variable:
org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore$$anonfun$asyncAddQueueMessage$1#1
	   Local Variable:
org.apache.activemq.store.leveldb.LevelDBPersistenceAdapter#1
	at
org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.asyncAddQueueMessage(LevelDBStore.scala:784)
	   Local Variable:
org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore#1
	at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:829)
	at org.apache.activemq.broker.region.Queue.send(Queue.java:723)
	   Local Variable: org.apache.activemq.broker.region.Queue#1
	   Local Variable: org.apache.activemq.state.ProducerState#2
	   Local Variable: org.apache.activemq.command.ProducerInfo#3
	at
org.apache.activemq.broker.region.DestinationFilter.send(DestinationFilter.java:132)
	   Local Variable:
org.apache.activemq.security.AuthorizationDestinationFilter#1
	at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:419)
	   Local Variable: org.apache.activemq.broker.jmx.ManagedQueueRegion#1
	at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:468)
	at
org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)
	   Local Variable: org.apache.activemq.broker.jmx.ManagedRegionBroker#1
	at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)
	   Local Variable: org.apache.activemq.advisory.AdvisoryBroker#1
	at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
	   Local Variable: org.apache.activemq.command.ActiveMQQueue#27
	   Local Variable: org.apache.activemq.broker.CompositeDestinationBroker#1
	at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:293)
	   Local Variable: org.apache.activemq.broker.ConnectionContext#3
	   Local Variable: org.apache.activemq.broker.TransactionBroker#1
	at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)
	   Local Variable: org.apache.activemq.security.JaasAuthenticationBroker#1
	at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)
	at
org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:223)
	   Local Variable:
org.apache.activemq.security.JaasAuthenticationBroker$JaasSecurityContext#2
	   Local Variable: org.apache.activemq.security.AuthorizationBroker#1
	at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)
	   Local Variable: org.apache.activemq.broker.BrokerService$5#1
	at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:542)
	   Local Variable: org.apache.activemq.broker.ProducerBrokerExchange#2
	   Local Variable: org.apache.activemq.command.ProducerId#39
	at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:768)
	at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:334)
	   Local Variable:
org.apache.activemq.broker.jmx.ManagedTransportConnection#2
	at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)
	   Local Variable: org.apache.activemq.broker.TransportConnection$1#2
	at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
	   Local Variable: org.apache.activemq.transport.MutexTransport#4
	at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
	   Local Variable: org.apache.activemq.transport.WireFormatNegotiator#4
	at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)
	   Local Variable: org.apache.activemq.transport.InactivityMonitor#4
	at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
	at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
	   Local Variable: org.apache.activemq.command.ActiveMQObjectMessage#27
	at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
	   Local Variable: org.apache.activemq.transport.tcp.TcpTransport#4
	at java.lang.Thread.run(Thread.java:745)

How can I improve the Store and forward?

Thanks a log again for your support.

Greetings

Michele







--
View this message in context: http://camel.465427.n5.nabble.com/Best-Strategy-to-process-a-large-number-of-rows-in-File-tp5779856p5780047.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Mime
View raw message