directory-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Carlo.Acco...@ibs-ag.com>
Subject Options to get a replica consumer initially loaded with large db
Date Wed, 06 Nov 2013 22:03:13 GMT
Hi,

We've successfully setup replication (m16 snapshot) and it's working great on our test instances
with ~10K entries.
With  ~85K entries, we cannot get a consumer (slave) instance initially loaded without running
out of memory.
We're on 32 bit so the most we can set the JVM memory to is ~ 1.34 GB

2 questions:
Is it possible to batch the replication changes 5000 at a time to give the jvm time to catch
up?
We boot up an empty instance and it starts replicating the 85k entries. It runs out of memory
before it's done.
We're not sure what the state of the database is once this occurs.
Again, when there are only 10k, it completes no problem. We tried setting ads-replSearchSizeLimit
to 5k but this doesn't seem to make any difference.

Also,  Is it possible to configure a consumer to begin replicating from a certain time?  For
example, if I wanted the consumer to
Pull everything in from 2 pm yesterday? We want to replay changes from a certain point to
the slaves.


Thanks,

Here's how our consumers are configured.

dn: ads-replConsumerId=cpro,ou=replConsumers,ads-serverId=ldapServer,ou=servers,ads-directoryServiceId=default,ou=config
objectClass: top
objectClass: ads-base
objectClass: ads-replConsumer
ads-replSearchSizeLimit: 5000
ads-replAttributes: *
ads-replConsumerId: cpro
ads-replRefreshInterval: 60000
ads-replUserPassword: secret
ads-replStrictCertValidation: FALSE
ads-replUserDn: uid=admin,ou=system
ads-replUseTls: FALSE
ads-replProvPort: 10389
ads-replProvHostName: localhost
ads-replRefreshNPersist: TRUE
ads-replSearchScope: sub
ads-replSearchTimeOut: 0
ads-searchBaseDN: o=cpro
ads-replSearchFilter: (objectClass=*)
ads-enabled: TRUE
ads-replAliasDerefMode: never

Java OOM
jvm 1    | Exception in thread "Thread-4"
jvm 1    | java.lang.OutOfMemoryError: Java heap space
jvm 1    |      at java.lang.Class.getDeclaredMethods0(Native Method)
jvm 1    |      at java.lang.Class.privateGetDeclaredMethods(Class.java:2436)
jvm 1    |      at java.lang.Class.getDeclaredMethod(Class.java:1937)
jvm 1    |      at java.io.ObjectStreamClass.getInheritableMethod(ObjectStreamClass.java:1344)
jvm 1    |      at java.io.ObjectStreamClass.access$2200(ObjectStreamClass.java:50)
jvm 1    |      at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:446)
jvm 1    |      at java.security.AccessController.doPrivileged(Native Method)
jvm 1    |      at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:411)
jvm 1    |      at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
jvm 1    |      at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
jvm 1    |      at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
jvm 1    |      at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
jvm 1    |      at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
jvm 1    |      at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
jvm 1    |      at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
jvm 1    |      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1114)
jvm 1    |      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
jvm 1    |      at jdbm.btree.BTree.writeExternal(BTree.java:580)
jvm 1    |      at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
jvm 1    |      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
jvm 1    |      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
jvm 1    |      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
jvm 1    |      at jdbm.helper.Serialization.serialize(Serialization.java:74)
jvm 1    |      at jdbm.helper.DefaultSerializer.serialize(DefaultSerializer.java:81)
jvm 1    |      at jdbm.recman.BaseRecordManager.update(BaseRecordManager.java:274)
jvm 1    |      at jdbm.recman.CacheRecordManager.updateCacheEntries(CacheRecordManager.java:417)
jvm 1    |      at jdbm.recman.CacheRecordManager.commit(CacheRecordManager.java:349)
jvm 1    |      at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.sync(JdbmTable.java:977)
jvm 1    |      at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.commit(JdbmTable.java:1183)
jvm 1    |      at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.remove(JdbmTable.java:829)
jvm 1    |      at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.drop(JdbmIndex.java:435)
jvm 1    |      at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.drop(JdbmIndex.java:1)
jvm 1    | Server daemon died!
jvm 1    | Exception in thread "pool-4-thread-1"
jvm 1    | java.lang.OutOfMemoryError: Java heap space
jvm 1    |      at java.util.HashMap.newKeyIterator(HashMap.java:840)
jvm 1    |      at java.util.HashMap$KeySet.iterator(HashMap.java:874)
jvm 1    |      at java.util.HashSet.iterator(HashSet.java:153)
jvm 1    |      at java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1005)
jvm 1    |      at java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1004)
jvm 1    |      at org.apache.mina.transport.socket.nio.NioProcessor$IoSessionIterator.<init>(NioProcessor.java:321)
jvm 1    |      at org.apache.mina.transport.socket.nio.NioProcessor$IoSessionIterator.<init>(NioProcessor.java:311)
jvm 1    |      at org.apache.mina.transport.socket.nio.NioProcessor.allSessions(NioProcessor.java:93)
jvm 1    |      at org.apache.mina.core.polling.AbstractPollingIoProcessor.notifyIdleSessions(AbstractPollingIoProcessor.java:760)
jvm 1    |      at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$900(AbstractPollingIoProcessor.java:67)
jvm 1    |      at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1135)
jvm 1    |      at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
jvm 1    |      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
jvm 1    |      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
jvm 1    |      at java.lang.Thread.run(Thread.java:662)
jvm 1    | java.lang.OutOfMemoryError: Java heap space
jvm 1    |      at java.lang.String.toCharArray(String.java:2725)
jvm 1    |      at sun.nio.cs.SingleByteDecoder.<init>(SingleByteDecoder.java:27)
jvm 1    |      at sun.nio.cs.MS1252$Decoder.<init>(MS1252.java:72)
jvm 1    |      at sun.nio.cs.MS1252.newDecoder(MS1252.java:39)
jvm 1    |      at java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:116)
jvm 1    |      at java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:108)
jvm 1    |      at java.lang.StringCoding.decode(StringCoding.java:167)
jvm 1    |      at java.lang.StringCoding.decode(StringCoding.java:185)
jvm 1    |      at java.lang.String.<init>(String.java:570)
jvm 1    |      at org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:3759)
jvm 1    |      at org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084)
jvm 1    |      at java.lang.Thread.run(Thread.java:662)
jvm 1    | [15:47:16] WARN [org.apache.directory.ldap.client.api.LdapNetworkConnection] -
Java heap space
jvm 1    | java.lang.OutOfMemoryError: Java heap space
wrapper  | JVM appears hung: Timed out waiting for signal from JVM.
wrapper  | JVM did not exit on request, terminated
wrapper  | Launching a JVM...
jvm 2    | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
jvm 2    |   Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.
jvm 2    |
jvm 2    | [15:54:56] ERROR [org.apache.directory.server.core.authz.GroupCache]
- Exception while initializing the groupCache:  {}
jvm 2    | org.apache.directory.api.ldap.model.exception.LdapOperationErrorException
jvm 2    |      at org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1120)
jvm 2    |      at org.apache.directory.server.xdbm.search.evaluator.EqualityEvaluator.evaluate(EqualityEvaluator.java:97)
jvm 2    |      at org.apache.directory.server.xdbm.search.evaluator.AndEvaluator.evaluate(AndEvaluator.java:110)
jvm 2    |      at org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:167)
jvm 2    |      at org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:1)
jvm 2    |      at org.apache.directory.server.core.api.filtering.BaseEntryFilteringCursor.next(BaseEntryFilteringCursor.java:377)
jvm 2    |      at org.apache.directory.server.core.authz.GroupCache.initialize(GroupCache.java:164)
jvm 2    |      at org.apache.directory.server.core.authz.GroupCache.<init>(GroupCache.java:122)
jvm 2    |      at org.apache.directory.server.core.authz.AciAuthorizationInterceptor.init(AciAuthorizationInterceptor.java:286)
jvm 2    |      at org.apache.directory.server.core.DefaultDirectoryService.initInterceptors(DefaultDirectoryService.java:688)
jvm 2    |      at org.apache.directory.server.core.DefaultDirectoryService.initialize(DefaultDirectoryService.java:1836)
jvm 2    |      at org.apache.directory.server.core.DefaultDirectoryService.startup(DefaultDirectoryService.java:1247)
jvm 2    |      at org.apache.directory.server.ApacheDsService.initDirectoryService(ApacheDsService.java:323)
jvm 2    |      at org.apache.directory.server.ApacheDsService.start(ApacheDsService.java:182)
jvm 2    |      at org.apache.directory.server.wrapper.ApacheDsTanukiWrapper.start(ApacheDsTanukiWrapper.java:72)
jvm 2    |      at org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:2788)
jvm 2    | Caused by: org.apache.directory.api.ldap.model.exception.LdapOperationErrorException
jvm 2    |      at org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1196)
jvm 2    |      at org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1116)
jvm 2    |      ... 15 more


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message