directory-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Carlo.Acco...@ibs-ag.com>
Subject RE: Options to get a replica consumer initially loaded with large db
Date Thu, 07 Nov 2013 17:20:12 GMT
Kiran, thanks.
Yes the out of memory issue is on consumer, the provider is fine and memory does not grow
hardly at all. 

On or around line 1148 of ServiceBuilder the SyncReplConfiguration size limit is not being
set from the replBean. 
When I add the line below 

                config.setSearchSizeLimit( replBean.getReplSearchSizeLimit() );


The consumer only processes 5000 entries. That takes care of the JVM OOM but the consumer
does not appear to repeatedly process another 5000 at a time. Should it, can it? 
Thanks.


-----Original Message-----
From: ayyagarikiran@gmail.com [mailto:ayyagarikiran@gmail.com] On Behalf Of Kiran Ayyagari
Sent: Wednesday, November 06, 2013 11:43 PM
To: users@directory.apache.org
Subject: Re: Options to get a replica consumer initially loaded with large db

Carlo,

On Thu, Nov 7, 2013 at 3:33 AM, <Carlo.Accorsi@ibs-ag.com> wrote:

> Hi,
>
> We've successfully setup replication (m16 snapshot) and it's working 
> great on our test instances with ~10K entries.
> With  ~85K entries, we cannot get a consumer (slave) instance 
> initially loaded without running out of memory.
>
this error is on consumer correct?

> We're on 32 bit so the most we can set the JVM memory to is ~ 1.34 GB
>
> 2 questions:
> Is it possible to batch the replication changes 5000 at a time to give 
> the jvm time to catch up?
>
should be possible with the config setting you mentioned below

> We boot up an empty instance and it starts replicating the 85k 
> entries. It runs out of memory before it's done.
> We're not sure what the state of the database is once this occurs.
> Again, when there are only 10k, it completes no problem. We tried 
> setting ads-replSearchSizeLimit to 5k but this doesn't seem to make any difference.
>
> this is strange, the value is indeed used while sending the 
> replication
rquest

> Also,  Is it possible to configure a consumer to begin replicating 
> from a certain time?  For example, if I wanted the consumer to Pull 
> everything in from 2 pm yesterday? We want to replay changes from a 
> certain point to the slaves.
>
> the server already takes care of this based on the previous state 
> stored
(using what is known as a 'cookie')
so after restart a consumer should be able to pickup from where it left

>
> Thanks,
>
> Here's how our consumers are configured.
>
> dn:
> ads-replConsumerId=cpro,ou=replConsumers,ads-serverId=ldapServer,ou=se
> rvers,ads-directoryServiceId=default,ou=config
> objectClass: top
> objectClass: ads-base
> objectClass: ads-replConsumer
> ads-replSearchSizeLimit: 5000
> ads-replAttributes: *
> ads-replConsumerId: cpro
> ads-replRefreshInterval: 60000
> ads-replUserPassword: secret
> ads-replStrictCertValidation: FALSE
> ads-replUserDn: uid=admin,ou=system
> ads-replUseTls: FALSE
> ads-replProvPort: 10389
> ads-replProvHostName: localhost
> ads-replRefreshNPersist: TRUE
> ads-replSearchScope: sub
> ads-replSearchTimeOut: 0
> ads-searchBaseDN: o=cpro
> ads-replSearchFilter: (objectClass=*)
> ads-enabled: TRUE
> ads-replAliasDerefMode: never
>
> Java OOM
> jvm 1    | Exception in thread "Thread-4"
> jvm 1    | java.lang.OutOfMemoryError: Java heap space
> jvm 1    |      at java.lang.Class.getDeclaredMethods0(Native Method)
> jvm 1    |      at
> java.lang.Class.privateGetDeclaredMethods(Class.java:2436)
> jvm 1    |      at java.lang.Class.getDeclaredMethod(Class.java:1937)
> jvm 1    |      at
> java.io.ObjectStreamClass.getInheritableMethod(ObjectStreamClass.java:1344)
> jvm 1    |      at
> java.io.ObjectStreamClass.access$2200(ObjectStreamClass.java:50)
> jvm 1    |      at
> java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:446)
> jvm 1    |      at java.security.AccessController.doPrivileged(Native
> Method)
> jvm 1    |      at
> java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:411)
> jvm 1    |      at
> java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> jvm 1    |      at
> java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
> jvm 1    |      at
> java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> jvm 1    |      at
> java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
> jvm 1    |      at
> java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> jvm 1    |      at
> java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
> jvm 1    |      at
> java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> jvm 1    |      at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1114)
> jvm 1    |      at
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
> jvm 1    |      at jdbm.btree.BTree.writeExternal(BTree.java:580)
> jvm 1    |      at
> java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
> jvm 1    |      at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
> jvm 1    |      at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
> jvm 1    |      at
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
> jvm 1    |      at
> jdbm.helper.Serialization.serialize(Serialization.java:74)
> jvm 1    |      at
> jdbm.helper.DefaultSerializer.serialize(DefaultSerializer.java:81)
> jvm 1    |      at
> jdbm.recman.BaseRecordManager.update(BaseRecordManager.java:274)
> jvm 1    |      at
> jdbm.recman.CacheRecordManager.updateCacheEntries(CacheRecordManager.java:417)
> jvm 1    |      at
> jdbm.recman.CacheRecordManager.commit(CacheRecordManager.java:349)
> jvm 1    |      at
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.sync(JdbmTable.java:977)
> jvm 1    |      at
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.commit(JdbmTable.java:1183)
> jvm 1    |      at
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.remove(JdbmTable.java:829)
> jvm 1    |      at
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.drop(JdbmIndex.java:435)
> jvm 1    |      at
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.drop(JdbmIndex.java:1)
> jvm 1    | Server daemon died!
>
this definitely looking like a JDBM cache related issue, can you file a bug please

> jvm 1    | Exception in thread "pool-4-thread-1"
> jvm 1    | java.lang.OutOfMemoryError: Java heap space
> jvm 1    |      at java.util.HashMap.newKeyIterator(HashMap.java:840)
> jvm 1    |      at java.util.HashMap$KeySet.iterator(HashMap.java:874)
> jvm 1    |      at java.util.HashSet.iterator(HashSet.java:153)
> jvm 1    |      at
> java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1005)
> jvm 1    |      at
> java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1004)
> jvm 1    |      at
> org.apache.mina.transport.socket.nio.NioProcessor$IoSessionIterator.<init>(NioProcessor.java:321)
> jvm 1    |      at
> org.apache.mina.transport.socket.nio.NioProcessor$IoSessionIterator.<init>(NioProcessor.java:311)
> jvm 1    |      at
> org.apache.mina.transport.socket.nio.NioProcessor.allSessions(NioProcessor.java:93)
> jvm 1    |      at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.notifyIdleSessions(AbstractPollingIoProcessor.java:760)
> jvm 1    |      at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.access$900(AbstractPollingIoProcessor.java:67)
> jvm 1    |      at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1135)
> jvm 1    |      at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> jvm 1    |      at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> jvm 1    |      at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> jvm 1    |      at java.lang.Thread.run(Thread.java:662)
> jvm 1    | java.lang.OutOfMemoryError: Java heap space
> jvm 1    |      at java.lang.String.toCharArray(String.java:2725)
> jvm 1    |      at
> sun.nio.cs.SingleByteDecoder.<init>(SingleByteDecoder.java:27)
> jvm 1    |      at sun.nio.cs.MS1252$Decoder.<init>(MS1252.java:72)
> jvm 1    |      at sun.nio.cs.MS1252.newDecoder(MS1252.java:39)
> jvm 1    |      at
> java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:116)
> jvm 1    |      at
> java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:108)
> jvm 1    |      at java.lang.StringCoding.decode(StringCoding.java:167)
> jvm 1    |      at java.lang.StringCoding.decode(StringCoding.java:185)
> jvm 1    |      at java.lang.String.<init>(String.java:570)
> jvm 1    |      at
> org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:3759)
> jvm 1    |      at
> org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084)
> jvm 1    |      at java.lang.Thread.run(Thread.java:662)
> jvm 1    | [15:47:16] WARN
> [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Java 
> heap space
> jvm 1    | java.lang.OutOfMemoryError: Java heap space
> wrapper  | JVM appears hung: Timed out waiting for signal from JVM.
> wrapper  | JVM did not exit on request, terminated wrapper  | 
> Launching a JVM...
> jvm 2    | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
> jvm 2    |   Copyright 1999-2006 Tanuki Software, Inc.  All Rights
> Reserved.
> jvm 2    |
>
is the below error thrown when a restart was attempted by you after the process was killed
with OOM?

> jvm 2    | [15:54:56] ERROR
> [org.apache.directory.server.core.authz.GroupCache]
> - Exception while initializing the groupCache:  {}
> jvm 2    |
> org.apache.directory.api.ldap.model.exception.LdapOperationErrorException
> jvm 2    |      at
> org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1120)
> jvm 2    |      at
> org.apache.directory.server.xdbm.search.evaluator.EqualityEvaluator.evaluate(EqualityEvaluator.java:97)
> jvm 2    |      at
> org.apache.directory.server.xdbm.search.evaluator.AndEvaluator.evaluate(AndEvaluator.java:110)
> jvm 2    |      at
> org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:167)
> jvm 2    |      at
> org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:1)
> jvm 2    |      at
> org.apache.directory.server.core.api.filtering.BaseEntryFilteringCursor.next(BaseEntryFilteringCursor.java:377)
> jvm 2    |      at
> org.apache.directory.server.core.authz.GroupCache.initialize(GroupCache.java:164)
> jvm 2    |      at
> org.apache.directory.server.core.authz.GroupCache.<init>(GroupCache.java:122)
> jvm 2    |      at
> org.apache.directory.server.core.authz.AciAuthorizationInterceptor.init(AciAuthorizationInterceptor.java:286)
> jvm 2    |      at
> org.apache.directory.server.core.DefaultDirectoryService.initInterceptors(DefaultDirectoryService.java:688)
> jvm 2    |      at
> org.apache.directory.server.core.DefaultDirectoryService.initialize(DefaultDirectoryService.java:1836)
> jvm 2    |      at
> org.apache.directory.server.core.DefaultDirectoryService.startup(DefaultDirectoryService.java:1247)
> jvm 2    |      at
> org.apache.directory.server.ApacheDsService.initDirectoryService(ApacheDsService.java:323)
> jvm 2    |      at
> org.apache.directory.server.ApacheDsService.start(ApacheDsService.java:182)
> jvm 2    |      at
> org.apache.directory.server.wrapper.ApacheDsTanukiWrapper.start(ApacheDsTanukiWrapper.java:72)
> jvm 2    |      at
> org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:2788)
> jvm 2    | Caused by:
> org.apache.directory.api.ldap.model.exception.LdapOperationErrorException
> jvm 2    |      at
> org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1196)
> jvm 2    |      at
> org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1116)
> jvm 2    |      ... 15 more
>
>


--
Kiran Ayyagari
http://keydap.com

Mime
View raw message