directory-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kiran Ayyagari <kayyag...@apache.org>
Subject Re: Options to get a replica consumer initially loaded with large db
Date Thu, 07 Nov 2013 19:10:54 GMT
On Fri, Nov 8, 2013 at 12:37 AM, <Carlo.Accorsi@ibs-ag.com> wrote:

> Kiran.. setting ads-replRefreshNPersist: FALSE allowed us to load the 85k
> entries at once!
>
> Should we just leave it false?  Not sure the difference.  Thanks!!
>
> setting 'true' sets up persistent search with the server, 'false' will
periodically query the server
for updates, you can choose either one of these.

> Also entered the JIRA .
> https://issues.apache.org/jira/browse/DIRSERVER-1914
>
>
> -----Original Message-----
> From: ayyagarikiran@gmail.com [mailto:ayyagarikiran@gmail.com] On Behalf
> Of Kiran Ayyagari
> Sent: Thursday, November 07, 2013 12:34 PM
> To: users@directory.apache.org
> Subject: Re: Options to get a replica consumer initially loaded with large
> db
>
> On Thu, Nov 7, 2013 at 10:50 PM, <Carlo.Accorsi@ibs-ag.com> wrote:
>
> > Kiran, thanks.
> > Yes the out of memory issue is on consumer, the provider is fine and
> > memory does not grow hardly at all.
> >
> > On or around line 1148 of ServiceBuilder the SyncReplConfiguration
> > size limit is not being set from the replBean.
> > When I add the line below
> >
> >                 config.setSearchSizeLimit(
> > replBean.getReplSearchSizeLimit() );
> >
> > ahhhhh this is a bad mistake, thanks for finding it
>
> >
> > The consumer only processes 5000 entries. That takes care of the JVM
> > OOM but the consumer does not appear to repeatedly process another
> > 5000 at a time. Should it, can it?
> >
> it should, can you file a bug please.
> otoh, change the mode to refreshOnly by setting ads-replRefreshNPersist to
> 'false' and try one all the 85k gets replicated you can change the value to
> 'true'
>
> ads-replRefreshNPersist: FALSE
>
> Thanks.
> >
> >
> > -----Original Message-----
> > From: ayyagarikiran@gmail.com [mailto:ayyagarikiran@gmail.com] On
> > Behalf Of Kiran Ayyagari
> > Sent: Wednesday, November 06, 2013 11:43 PM
> > To: users@directory.apache.org
> > Subject: Re: Options to get a replica consumer initially loaded with
> > large db
> >
> > Carlo,
> >
> > On Thu, Nov 7, 2013 at 3:33 AM, <Carlo.Accorsi@ibs-ag.com> wrote:
> >
> > > Hi,
> > >
> > > We've successfully setup replication (m16 snapshot) and it's working
> > > great on our test instances with ~10K entries.
> > > With  ~85K entries, we cannot get a consumer (slave) instance
> > > initially loaded without running out of memory.
> > >
> > this error is on consumer correct?
> >
> > > We're on 32 bit so the most we can set the JVM memory to is ~ 1.34
> > > GB
> > >
> > > 2 questions:
> > > Is it possible to batch the replication changes 5000 at a time to
> > > give the jvm time to catch up?
> > >
> > should be possible with the config setting you mentioned below
> >
> > > We boot up an empty instance and it starts replicating the 85k
> > > entries. It runs out of memory before it's done.
> > > We're not sure what the state of the database is once this occurs.
> > > Again, when there are only 10k, it completes no problem. We tried
> > > setting ads-replSearchSizeLimit to 5k but this doesn't seem to make
> > > any
> > difference.
> > >
> > > this is strange, the value is indeed used while sending the
> > > replication
> > rquest
> >
> > > Also,  Is it possible to configure a consumer to begin replicating
> > > from a certain time?  For example, if I wanted the consumer to Pull
> > > everything in from 2 pm yesterday? We want to replay changes from a
> > > certain point to the slaves.
> > >
> > > the server already takes care of this based on the previous state
> > > stored
> > (using what is known as a 'cookie')
> > so after restart a consumer should be able to pickup from where it
> > left
> >
> > >
> > > Thanks,
> > >
> > > Here's how our consumers are configured.
> > >
> > > dn:
> > > ads-replConsumerId=cpro,ou=replConsumers,ads-serverId=ldapServer,ou=
> > > se rvers,ads-directoryServiceId=default,ou=config
> > > objectClass: top
> > > objectClass: ads-base
> > > objectClass: ads-replConsumer
> > > ads-replSearchSizeLimit: 5000
> > > ads-replAttributes: *
> > > ads-replConsumerId: cpro
> > > ads-replRefreshInterval: 60000
> > > ads-replUserPassword: secret
> > > ads-replStrictCertValidation: FALSE
> > > ads-replUserDn: uid=admin,ou=system
> > > ads-replUseTls: FALSE
> > > ads-replProvPort: 10389
> > > ads-replProvHostName: localhost
> > > ads-replRefreshNPersist: TRUE
> > > ads-replSearchScope: sub
> > > ads-replSearchTimeOut: 0
> > > ads-searchBaseDN: o=cpro
> > > ads-replSearchFilter: (objectClass=*)
> > > ads-enabled: TRUE
> > > ads-replAliasDerefMode: never
> > >
> > > Java OOM
> > > jvm 1    | Exception in thread "Thread-4"
> > > jvm 1    | java.lang.OutOfMemoryError: Java heap space
> > > jvm 1    |      at java.lang.Class.getDeclaredMethods0(Native Method)
> > > jvm 1    |      at
> > > java.lang.Class.privateGetDeclaredMethods(Class.java:2436)
> > > jvm 1    |      at java.lang.Class.getDeclaredMethod(Class.java:1937)
> > > jvm 1    |      at
> > >
> > java.io.ObjectStreamClass.getInheritableMethod(ObjectStreamClass.java:
> > 1344)
> > > jvm 1    |      at
> > > java.io.ObjectStreamClass.access$2200(ObjectStreamClass.java:50)
> > > jvm 1    |      at
> > > java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:446)
> > > jvm 1    |      at java.security.AccessController.doPrivileged(Native
> > > Method)
> > > jvm 1    |      at
> > > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:411)
> > > jvm 1    |      at
> > > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> > > jvm 1    |      at
> > > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
> > > jvm 1    |      at
> > > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> > > jvm 1    |      at
> > > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
> > > jvm 1    |      at
> > > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> > > jvm 1    |      at
> > > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
> > > jvm 1    |      at
> > > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> > > jvm 1    |      at
> > > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1114)
> > > jvm 1    |      at
> > > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
> > > jvm 1    |      at jdbm.btree.BTree.writeExternal(BTree.java:580)
> > > jvm 1    |      at
> > >
> > java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1
> > 429)
> > > jvm 1    |      at
> > >
> > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java
> > :1398)
> > > jvm 1    |      at
> > > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
> > > jvm 1    |      at
> > > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
> > > jvm 1    |      at
> > > jdbm.helper.Serialization.serialize(Serialization.java:74)
> > > jvm 1    |      at
> > > jdbm.helper.DefaultSerializer.serialize(DefaultSerializer.java:81)
> > > jvm 1    |      at
> > > jdbm.recman.BaseRecordManager.update(BaseRecordManager.java:274)
> > > jvm 1    |      at
> > >
> > jdbm.recman.CacheRecordManager.updateCacheEntries(CacheRecordManager.j
> > ava:417)
> > > jvm 1    |      at
> > > jdbm.recman.CacheRecordManager.commit(CacheRecordManager.java:349)
> > > jvm 1    |      at
> > >
> > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.s
> > ync(JdbmTable.java:977)
> > > jvm 1    |      at
> > >
> > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.c
> > ommit(JdbmTable.java:1183)
> > > jvm 1    |      at
> > >
> > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.r
> > emove(JdbmTable.java:829)
> > > jvm 1    |      at
> > >
> > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.d
> > rop(JdbmIndex.java:435)
> > > jvm 1    |      at
> > >
> > org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.d
> > rop(JdbmIndex.java:1)
> > > jvm 1    | Server daemon died!
> > >
> > this definitely looking like a JDBM cache related issue, can you file
> > a bug please
> >
> > > jvm 1    | Exception in thread "pool-4-thread-1"
> > > jvm 1    | java.lang.OutOfMemoryError: Java heap space
> > > jvm 1    |      at java.util.HashMap.newKeyIterator(HashMap.java:840)
> > > jvm 1    |      at java.util.HashMap$KeySet.iterator(HashMap.java:874)
> > > jvm 1    |      at java.util.HashSet.iterator(HashSet.java:153)
> > > jvm 1    |      at
> > >
> > java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java
> > :1005)
> > > jvm 1    |      at
> > >
> > java.util.Collections$UnmodifiableCollection.iterator(Collections.java
> > :1004)
> > > jvm 1    |      at
> > >
> > org.apache.mina.transport.socket.nio.NioProcessor$IoSessionIterator.<i
> > nit>(NioProcessor.java:321)
> > > jvm 1    |      at
> > >
> > org.apache.mina.transport.socket.nio.NioProcessor$IoSessionIterator.<i
> > nit>(NioProcessor.java:311)
> > > jvm 1    |      at
> > >
> > org.apache.mina.transport.socket.nio.NioProcessor.allSessions(NioProce
> > ssor.java:93)
> > > jvm 1    |      at
> > >
> > org.apache.mina.core.polling.AbstractPollingIoProcessor.notifyIdleSess
> > ions(AbstractPollingIoProcessor.java:760)
> > > jvm 1    |      at
> > >
> > org.apache.mina.core.polling.AbstractPollingIoProcessor.access$900(Abs
> > tractPollingIoProcessor.java:67)
> > > jvm 1    |      at
> > >
> > org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(
> > AbstractPollingIoProcessor.java:1135)
> > > jvm 1    |      at
> > >
> > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable
> > .java:64)
> > > jvm 1    |      at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu
> > tor.java:895)
> > > jvm 1    |      at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> > java:918)
> > > jvm 1    |      at java.lang.Thread.run(Thread.java:662)
> > > jvm 1    | java.lang.OutOfMemoryError: Java heap space
> > > jvm 1    |      at java.lang.String.toCharArray(String.java:2725)
> > > jvm 1    |      at
> > > sun.nio.cs.SingleByteDecoder.<init>(SingleByteDecoder.java:27)
> > > jvm 1    |      at sun.nio.cs.MS1252$Decoder.<init>(MS1252.java:72)
> > > jvm 1    |      at sun.nio.cs.MS1252.newDecoder(MS1252.java:39)
> > > jvm 1    |      at
> > > java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:116)
> > > jvm 1    |      at
> > > java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:108)
> > > jvm 1    |      at java.lang.StringCoding.decode(StringCoding.java:167)
> > > jvm 1    |      at java.lang.StringCoding.decode(StringCoding.java:185)
> > > jvm 1    |      at java.lang.String.<init>(String.java:570)
> > > jvm 1    |      at
> > >
> > org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.
> > java:3759)
> > > jvm 1    |      at
> > > org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084)
> > > jvm 1    |      at java.lang.Thread.run(Thread.java:662)
> > > jvm 1    | [15:47:16] WARN
> > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Java
> > > heap space
> > > jvm 1    | java.lang.OutOfMemoryError: Java heap space
> > > wrapper  | JVM appears hung: Timed out waiting for signal from JVM.
> > > wrapper  | JVM did not exit on request, terminated wrapper  |
> > > Launching a JVM...
> > > jvm 2    | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
> > > jvm 2    |   Copyright 1999-2006 Tanuki Software, Inc.  All Rights
> > > Reserved.
> > > jvm 2    |
> > >
> > is the below error thrown when a restart was attempted by you after
> > the process was killed with OOM?
> >
> > > jvm 2    | [15:54:56] ERROR
> > > [org.apache.directory.server.core.authz.GroupCache]
> > > - Exception while initializing the groupCache:  {}
> > > jvm 2    |
> > >
> org.apache.directory.api.ldap.model.exception.LdapOperationErrorException
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.core.partition.impl.btree.AbstractBTreePar
> > tition.fetch(AbstractBTreePartition.java:1120)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.xdbm.search.evaluator.EqualityEvaluator.ev
> > aluate(EqualityEvaluator.java:97)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.xdbm.search.evaluator.AndEvaluator.evaluat
> > e(AndEvaluator.java:110)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.core.partition.impl.btree.EntryCursorAdapt
> > or.get(EntryCursorAdaptor.java:167)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.core.partition.impl.btree.EntryCursorAdapt
> > or.get(EntryCursorAdaptor.java:1)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.core.api.filtering.BaseEntryFilteringCurso
> > r.next(BaseEntryFilteringCursor.java:377)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.core.authz.GroupCache.initialize(GroupCach
> > e.java:164)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.core.authz.GroupCache.<init>(GroupCache.ja
> > va:122)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.core.authz.AciAuthorizationInterceptor.ini
> > t(AciAuthorizationInterceptor.java:286)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.core.DefaultDirectoryService.initIntercept
> > ors(DefaultDirectoryService.java:688)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.core.DefaultDirectoryService.initialize(De
> > faultDirectoryService.java:1836)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.core.DefaultDirectoryService.startup(Defau
> > ltDirectoryService.java:1247)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.ApacheDsService.initDirectoryService(Apach
> > eDsService.java:323)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.ApacheDsService.start(ApacheDsService.java
> > :182)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.wrapper.ApacheDsTanukiWrapper.start(Apache
> > DsTanukiWrapper.java:72)
> > > jvm 2    |      at
> > >
> > org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:2
> > 788)
> > > jvm 2    | Caused by:
> > >
> org.apache.directory.api.ldap.model.exception.LdapOperationErrorException
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.core.partition.impl.btree.AbstractBTreePar
> > tition.fetch(AbstractBTreePartition.java:1196)
> > > jvm 2    |      at
> > >
> > org.apache.directory.server.core.partition.impl.btree.AbstractBTreePar
> > tition.fetch(AbstractBTreePartition.java:1116)
> > > jvm 2    |      ... 15 more
> > >
> > >
> >
> >
> > --
> > Kiran Ayyagari
> > http://keydap.com
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>



-- 
Kiran Ayyagari
http://keydap.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message