directory-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kiran Ayyagari <kayyag...@apache.org>
Subject Re: Options to get a replica consumer initially loaded with large db
Date Thu, 07 Nov 2013 04:43:08 GMT
Carlo,

On Thu, Nov 7, 2013 at 3:33 AM, <Carlo.Accorsi@ibs-ag.com> wrote:

> Hi,
>
> We've successfully setup replication (m16 snapshot) and it's working great
> on our test instances with ~10K entries.
> With  ~85K entries, we cannot get a consumer (slave) instance initially
> loaded without running out of memory.
>
this error is on consumer correct?

> We're on 32 bit so the most we can set the JVM memory to is ~ 1.34 GB
>
> 2 questions:
> Is it possible to batch the replication changes 5000 at a time to give the
> jvm time to catch up?
>
should be possible with the config setting you mentioned below

> We boot up an empty instance and it starts replicating the 85k entries. It
> runs out of memory before it's done.
> We're not sure what the state of the database is once this occurs.
> Again, when there are only 10k, it completes no problem. We tried setting
> ads-replSearchSizeLimit to 5k but this doesn't seem to make any difference.
>
> this is strange, the value is indeed used while sending the replication
rquest

> Also,  Is it possible to configure a consumer to begin replicating from a
> certain time?  For example, if I wanted the consumer to
> Pull everything in from 2 pm yesterday? We want to replay changes from a
> certain point to the slaves.
>
> the server already takes care of this based on the previous state stored
(using what is known as a 'cookie')
so after restart a consumer should be able to pickup from where it left

>
> Thanks,
>
> Here's how our consumers are configured.
>
> dn:
> ads-replConsumerId=cpro,ou=replConsumers,ads-serverId=ldapServer,ou=servers,ads-directoryServiceId=default,ou=config
> objectClass: top
> objectClass: ads-base
> objectClass: ads-replConsumer
> ads-replSearchSizeLimit: 5000
> ads-replAttributes: *
> ads-replConsumerId: cpro
> ads-replRefreshInterval: 60000
> ads-replUserPassword: secret
> ads-replStrictCertValidation: FALSE
> ads-replUserDn: uid=admin,ou=system
> ads-replUseTls: FALSE
> ads-replProvPort: 10389
> ads-replProvHostName: localhost
> ads-replRefreshNPersist: TRUE
> ads-replSearchScope: sub
> ads-replSearchTimeOut: 0
> ads-searchBaseDN: o=cpro
> ads-replSearchFilter: (objectClass=*)
> ads-enabled: TRUE
> ads-replAliasDerefMode: never
>
> Java OOM
> jvm 1    | Exception in thread "Thread-4"
> jvm 1    | java.lang.OutOfMemoryError: Java heap space
> jvm 1    |      at java.lang.Class.getDeclaredMethods0(Native Method)
> jvm 1    |      at
> java.lang.Class.privateGetDeclaredMethods(Class.java:2436)
> jvm 1    |      at java.lang.Class.getDeclaredMethod(Class.java:1937)
> jvm 1    |      at
> java.io.ObjectStreamClass.getInheritableMethod(ObjectStreamClass.java:1344)
> jvm 1    |      at
> java.io.ObjectStreamClass.access$2200(ObjectStreamClass.java:50)
> jvm 1    |      at
> java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:446)
> jvm 1    |      at java.security.AccessController.doPrivileged(Native
> Method)
> jvm 1    |      at
> java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:411)
> jvm 1    |      at
> java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> jvm 1    |      at
> java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
> jvm 1    |      at
> java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> jvm 1    |      at
> java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
> jvm 1    |      at
> java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> jvm 1    |      at
> java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
> jvm 1    |      at
> java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> jvm 1    |      at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1114)
> jvm 1    |      at
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
> jvm 1    |      at jdbm.btree.BTree.writeExternal(BTree.java:580)
> jvm 1    |      at
> java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1429)
> jvm 1    |      at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1398)
> jvm 1    |      at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
> jvm 1    |      at
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
> jvm 1    |      at
> jdbm.helper.Serialization.serialize(Serialization.java:74)
> jvm 1    |      at
> jdbm.helper.DefaultSerializer.serialize(DefaultSerializer.java:81)
> jvm 1    |      at
> jdbm.recman.BaseRecordManager.update(BaseRecordManager.java:274)
> jvm 1    |      at
> jdbm.recman.CacheRecordManager.updateCacheEntries(CacheRecordManager.java:417)
> jvm 1    |      at
> jdbm.recman.CacheRecordManager.commit(CacheRecordManager.java:349)
> jvm 1    |      at
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.sync(JdbmTable.java:977)
> jvm 1    |      at
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.commit(JdbmTable.java:1183)
> jvm 1    |      at
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.remove(JdbmTable.java:829)
> jvm 1    |      at
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.drop(JdbmIndex.java:435)
> jvm 1    |      at
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.drop(JdbmIndex.java:1)
> jvm 1    | Server daemon died!
>
this definitely looking like a JDBM cache related issue, can you file a bug
please

> jvm 1    | Exception in thread "pool-4-thread-1"
> jvm 1    | java.lang.OutOfMemoryError: Java heap space
> jvm 1    |      at java.util.HashMap.newKeyIterator(HashMap.java:840)
> jvm 1    |      at java.util.HashMap$KeySet.iterator(HashMap.java:874)
> jvm 1    |      at java.util.HashSet.iterator(HashSet.java:153)
> jvm 1    |      at
> java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1005)
> jvm 1    |      at
> java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1004)
> jvm 1    |      at
> org.apache.mina.transport.socket.nio.NioProcessor$IoSessionIterator.<init>(NioProcessor.java:321)
> jvm 1    |      at
> org.apache.mina.transport.socket.nio.NioProcessor$IoSessionIterator.<init>(NioProcessor.java:311)
> jvm 1    |      at
> org.apache.mina.transport.socket.nio.NioProcessor.allSessions(NioProcessor.java:93)
> jvm 1    |      at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.notifyIdleSessions(AbstractPollingIoProcessor.java:760)
> jvm 1    |      at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.access$900(AbstractPollingIoProcessor.java:67)
> jvm 1    |      at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1135)
> jvm 1    |      at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> jvm 1    |      at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> jvm 1    |      at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> jvm 1    |      at java.lang.Thread.run(Thread.java:662)
> jvm 1    | java.lang.OutOfMemoryError: Java heap space
> jvm 1    |      at java.lang.String.toCharArray(String.java:2725)
> jvm 1    |      at
> sun.nio.cs.SingleByteDecoder.<init>(SingleByteDecoder.java:27)
> jvm 1    |      at sun.nio.cs.MS1252$Decoder.<init>(MS1252.java:72)
> jvm 1    |      at sun.nio.cs.MS1252.newDecoder(MS1252.java:39)
> jvm 1    |      at
> java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:116)
> jvm 1    |      at
> java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:108)
> jvm 1    |      at java.lang.StringCoding.decode(StringCoding.java:167)
> jvm 1    |      at java.lang.StringCoding.decode(StringCoding.java:185)
> jvm 1    |      at java.lang.String.<init>(String.java:570)
> jvm 1    |      at
> org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:3759)
> jvm 1    |      at
> org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084)
> jvm 1    |      at java.lang.Thread.run(Thread.java:662)
> jvm 1    | [15:47:16] WARN
> [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Java heap
> space
> jvm 1    | java.lang.OutOfMemoryError: Java heap space
> wrapper  | JVM appears hung: Timed out waiting for signal from JVM.
> wrapper  | JVM did not exit on request, terminated
> wrapper  | Launching a JVM...
> jvm 2    | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
> jvm 2    |   Copyright 1999-2006 Tanuki Software, Inc.  All Rights
> Reserved.
> jvm 2    |
>
is the below error thrown when a restart was attempted by you after the
process was killed with OOM?

> jvm 2    | [15:54:56] ERROR
> [org.apache.directory.server.core.authz.GroupCache]
> - Exception while initializing the groupCache:  {}
> jvm 2    |
> org.apache.directory.api.ldap.model.exception.LdapOperationErrorException
> jvm 2    |      at
> org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1120)
> jvm 2    |      at
> org.apache.directory.server.xdbm.search.evaluator.EqualityEvaluator.evaluate(EqualityEvaluator.java:97)
> jvm 2    |      at
> org.apache.directory.server.xdbm.search.evaluator.AndEvaluator.evaluate(AndEvaluator.java:110)
> jvm 2    |      at
> org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:167)
> jvm 2    |      at
> org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:1)
> jvm 2    |      at
> org.apache.directory.server.core.api.filtering.BaseEntryFilteringCursor.next(BaseEntryFilteringCursor.java:377)
> jvm 2    |      at
> org.apache.directory.server.core.authz.GroupCache.initialize(GroupCache.java:164)
> jvm 2    |      at
> org.apache.directory.server.core.authz.GroupCache.<init>(GroupCache.java:122)
> jvm 2    |      at
> org.apache.directory.server.core.authz.AciAuthorizationInterceptor.init(AciAuthorizationInterceptor.java:286)
> jvm 2    |      at
> org.apache.directory.server.core.DefaultDirectoryService.initInterceptors(DefaultDirectoryService.java:688)
> jvm 2    |      at
> org.apache.directory.server.core.DefaultDirectoryService.initialize(DefaultDirectoryService.java:1836)
> jvm 2    |      at
> org.apache.directory.server.core.DefaultDirectoryService.startup(DefaultDirectoryService.java:1247)
> jvm 2    |      at
> org.apache.directory.server.ApacheDsService.initDirectoryService(ApacheDsService.java:323)
> jvm 2    |      at
> org.apache.directory.server.ApacheDsService.start(ApacheDsService.java:182)
> jvm 2    |      at
> org.apache.directory.server.wrapper.ApacheDsTanukiWrapper.start(ApacheDsTanukiWrapper.java:72)
> jvm 2    |      at
> org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:2788)
> jvm 2    | Caused by:
> org.apache.directory.api.ldap.model.exception.LdapOperationErrorException
> jvm 2    |      at
> org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1196)
> jvm 2    |      at
> org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.fetch(AbstractBTreePartition.java:1116)
> jvm 2    |      ... 15 more
>
>


-- 
Kiran Ayyagari
http://keydap.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message