Mailing-List: contact users-help@directory.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@directory.apache.org
Received-SPF: pass (athena.apache.org: local policy)
From: <Carlo.Accorsi@ibs-ag.com>
To: <users@directory.apache.org>
Date: Thu, 7 Nov 2013 20:07:14 +0100
Subject: RE: Options to get a replica consumer initially loaded with large db
Thread-Topic: Options to get a replica consumer initially loaded with large
 db
Thread-Index: Ac7b34qF9d1QhD2jSxW+KEfzeW8kSwABrQLw
Message-ID: <2BE7E81B77921F43A6A273C2DF2FA6A43D01423DAF@IBSMBX.ibs-ag.com>
References: <2BE7E81B77921F43A6A273C2DF2FA6A43CE56F6EB5@IBSMBX.ibs-ag.com>
	<CABzFU-dOaT7ZXBTPwVS_SiUcKjpNb7PPTv=QPcuXbnBHsGpdXA@mail.gmail.com>
	<2BE7E81B77921F43A6A273C2DF2FA6A43D01423D60@IBSMBX.ibs-ag.com>
 <CABzFU-emM4W=Q4j+5gqPEcptzY3zfFi6G+ziZ8ZaTnGrj2sETQ@mail.gmail.com>
In-Reply-To: 
 <CABzFU-emM4W=Q4j+5gqPEcptzY3zfFi6G+ziZ8ZaTnGrj2sETQ@mail.gmail.com>
Accept-Language: en-US, de-DE
Content-Language: en-US
acceptlanguage: en-US, de-DE
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Kiran.. setting ads-replRefreshNPersist: FALSE allowed us to load the 85k e=
ntries at once! =20

Should we just leave it false?  Not sure the difference.  Thanks!!

Also entered the JIRA .=20
https://issues.apache.org/jira/browse/DIRSERVER-1914


-----Original Message-----
From: ayyagarikiran@gmail.com [mailto:ayyagarikiran@gmail.com] On Behalf Of=
 Kiran Ayyagari
Sent: Thursday, November 07, 2013 12:34 PM
To: users@directory.apache.org
Subject: Re: Options to get a replica consumer initially loaded with large =
db

On Thu, Nov 7, 2013 at 10:50 PM, <Carlo.Accorsi@ibs-ag.com> wrote:

> Kiran, thanks.
> Yes the out of memory issue is on consumer, the provider is fine and=20
> memory does not grow hardly at all.
>
> On or around line 1148 of ServiceBuilder the SyncReplConfiguration=20
> size limit is not being set from the replBean.
> When I add the line below
>
>                 config.setSearchSizeLimit(
> replBean.getReplSearchSizeLimit() );
>
> ahhhhh this is a bad mistake, thanks for finding it

>
> The consumer only processes 5000 entries. That takes care of the JVM=20
> OOM but the consumer does not appear to repeatedly process another=20
> 5000 at a time. Should it, can it?
>
it should, can you file a bug please.
otoh, change the mode to refreshOnly by setting ads-replRefreshNPersist to =
'false' and try one all the 85k gets replicated you can change the value to=
 'true'

ads-replRefreshNPersist: FALSE

Thanks.
>
>
> -----Original Message-----
> From: ayyagarikiran@gmail.com [mailto:ayyagarikiran@gmail.com] On=20
> Behalf Of Kiran Ayyagari
> Sent: Wednesday, November 06, 2013 11:43 PM
> To: users@directory.apache.org
> Subject: Re: Options to get a replica consumer initially loaded with=20
> large db
>
> Carlo,
>
> On Thu, Nov 7, 2013 at 3:33 AM, <Carlo.Accorsi@ibs-ag.com> wrote:
>
> > Hi,
> >
> > We've successfully setup replication (m16 snapshot) and it's working=20
> > great on our test instances with ~10K entries.
> > With  ~85K entries, we cannot get a consumer (slave) instance=20
> > initially loaded without running out of memory.
> >
> this error is on consumer correct?
>
> > We're on 32 bit so the most we can set the JVM memory to is ~ 1.34=20
> > GB
> >
> > 2 questions:
> > Is it possible to batch the replication changes 5000 at a time to=20
> > give the jvm time to catch up?
> >
> should be possible with the config setting you mentioned below
>
> > We boot up an empty instance and it starts replicating the 85k=20
> > entries. It runs out of memory before it's done.
> > We're not sure what the state of the database is once this occurs.
> > Again, when there are only 10k, it completes no problem. We tried=20
> > setting ads-replSearchSizeLimit to 5k but this doesn't seem to make=20
> > any
> difference.
> >
> > this is strange, the value is indeed used while sending the=20
> > replication
> rquest
>
> > Also,  Is it possible to configure a consumer to begin replicating=20
> > from a certain time?  For example, if I wanted the consumer to Pull=20
> > everything in from 2 pm yesterday? We want to replay changes from a=20
> > certain point to the slaves.
> >
> > the server already takes care of this based on the previous state=20
> > stored
> (using what is known as a 'cookie')
> so after restart a consumer should be able to pickup from where it=20
> left
>
> >
> > Thanks,
> >
> > Here's how our consumers are configured.
> >
> > dn:
> > ads-replConsumerId=3Dcpro,ou=3DreplConsumers,ads-serverId=3DldapServer,=
ou=3D
> > se rvers,ads-directoryServiceId=3Ddefault,ou=3Dconfig
> > objectClass: top
> > objectClass: ads-base
> > objectClass: ads-replConsumer
> > ads-replSearchSizeLimit: 5000
> > ads-replAttributes: *
> > ads-replConsumerId: cpro
> > ads-replRefreshInterval: 60000
> > ads-replUserPassword: secret
> > ads-replStrictCertValidation: FALSE
> > ads-replUserDn: uid=3Dadmin,ou=3Dsystem
> > ads-replUseTls: FALSE
> > ads-replProvPort: 10389
> > ads-replProvHostName: localhost
> > ads-replRefreshNPersist: TRUE
> > ads-replSearchScope: sub
> > ads-replSearchTimeOut: 0
> > ads-searchBaseDN: o=3Dcpro
> > ads-replSearchFilter: (objectClass=3D*)
> > ads-enabled: TRUE
> > ads-replAliasDerefMode: never
> >
> > Java OOM
> > jvm 1    | Exception in thread "Thread-4"
> > jvm 1    | java.lang.OutOfMemoryError: Java heap space
> > jvm 1    |      at java.lang.Class.getDeclaredMethods0(Native Method)
> > jvm 1    |      at
> > java.lang.Class.privateGetDeclaredMethods(Class.java:2436)
> > jvm 1    |      at java.lang.Class.getDeclaredMethod(Class.java:1937)
> > jvm 1    |      at
> >
> java.io.ObjectStreamClass.getInheritableMethod(ObjectStreamClass.java:
> 1344)
> > jvm 1    |      at
> > java.io.ObjectStreamClass.access$2200(ObjectStreamClass.java:50)
> > jvm 1    |      at
> > java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:446)
> > jvm 1    |      at java.security.AccessController.doPrivileged(Native
> > Method)
> > jvm 1    |      at
> > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:411)
> > jvm 1    |      at
> > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> > jvm 1    |      at
> > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
> > jvm 1    |      at
> > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> > jvm 1    |      at
> > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
> > jvm 1    |      at
> > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> > jvm 1    |      at
> > java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:407)
> > jvm 1    |      at
> > java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:308)
> > jvm 1    |      at
> > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1114)
> > jvm 1    |      at
> > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
> > jvm 1    |      at jdbm.btree.BTree.writeExternal(BTree.java:580)
> > jvm 1    |      at
> >
> java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1
> 429)
> > jvm 1    |      at
> >
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java
> :1398)
> > jvm 1    |      at
> > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
> > jvm 1    |      at
> > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
> > jvm 1    |      at
> > jdbm.helper.Serialization.serialize(Serialization.java:74)
> > jvm 1    |      at
> > jdbm.helper.DefaultSerializer.serialize(DefaultSerializer.java:81)
> > jvm 1    |      at
> > jdbm.recman.BaseRecordManager.update(BaseRecordManager.java:274)
> > jvm 1    |      at
> >
> jdbm.recman.CacheRecordManager.updateCacheEntries(CacheRecordManager.j
> ava:417)
> > jvm 1    |      at
> > jdbm.recman.CacheRecordManager.commit(CacheRecordManager.java:349)
> > jvm 1    |      at
> >
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.s
> ync(JdbmTable.java:977)
> > jvm 1    |      at
> >
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.c
> ommit(JdbmTable.java:1183)
> > jvm 1    |      at
> >
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.r
> emove(JdbmTable.java:829)
> > jvm 1    |      at
> >
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.d
> rop(JdbmIndex.java:435)
> > jvm 1    |      at
> >
> org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmIndex.d
> rop(JdbmIndex.java:1)
> > jvm 1    | Server daemon died!
> >
> this definitely looking like a JDBM cache related issue, can you file=20
> a bug please
>
> > jvm 1    | Exception in thread "pool-4-thread-1"
> > jvm 1    | java.lang.OutOfMemoryError: Java heap space
> > jvm 1    |      at java.util.HashMap.newKeyIterator(HashMap.java:840)
> > jvm 1    |      at java.util.HashMap$KeySet.iterator(HashMap.java:874)
> > jvm 1    |      at java.util.HashSet.iterator(HashSet.java:153)
> > jvm 1    |      at
> >
> java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java
> :1005)
> > jvm 1    |      at
> >
> java.util.Collections$UnmodifiableCollection.iterator(Collections.java
> :1004)
> > jvm 1    |      at
> >
> org.apache.mina.transport.socket.nio.NioProcessor$IoSessionIterator.<i
> nit>(NioProcessor.java:321)
> > jvm 1    |      at
> >
> org.apache.mina.transport.socket.nio.NioProcessor$IoSessionIterator.<i
> nit>(NioProcessor.java:311)
> > jvm 1    |      at
> >
> org.apache.mina.transport.socket.nio.NioProcessor.allSessions(NioProce
> ssor.java:93)
> > jvm 1    |      at
> >
> org.apache.mina.core.polling.AbstractPollingIoProcessor.notifyIdleSess
> ions(AbstractPollingIoProcessor.java:760)
> > jvm 1    |      at
> >
> org.apache.mina.core.polling.AbstractPollingIoProcessor.access$900(Abs
> tractPollingIoProcessor.java:67)
> > jvm 1    |      at
> >
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(
> AbstractPollingIoProcessor.java:1135)
> > jvm 1    |      at
> >
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable
> .java:64)
> > jvm 1    |      at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu
> tor.java:895)
> > jvm 1    |      at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> java:918)
> > jvm 1    |      at java.lang.Thread.run(Thread.java:662)
> > jvm 1    | java.lang.OutOfMemoryError: Java heap space
> > jvm 1    |      at java.lang.String.toCharArray(String.java:2725)
> > jvm 1    |      at
> > sun.nio.cs.SingleByteDecoder.<init>(SingleByteDecoder.java:27)
> > jvm 1    |      at sun.nio.cs.MS1252$Decoder.<init>(MS1252.java:72)
> > jvm 1    |      at sun.nio.cs.MS1252.newDecoder(MS1252.java:39)
> > jvm 1    |      at
> > java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:116)
> > jvm 1    |      at
> > java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:108)
> > jvm 1    |      at java.lang.StringCoding.decode(StringCoding.java:167)
> > jvm 1    |      at java.lang.StringCoding.decode(StringCoding.java:185)
> > jvm 1    |      at java.lang.String.<init>(String.java:570)
> > jvm 1    |      at
> >
> org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.
> java:3759)
> > jvm 1    |      at
> > org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084)
> > jvm 1    |      at java.lang.Thread.run(Thread.java:662)
> > jvm 1    | [15:47:16] WARN
> > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Java=20
> > heap space
> > jvm 1    | java.lang.OutOfMemoryError: Java heap space
> > wrapper  | JVM appears hung: Timed out waiting for signal from JVM.
> > wrapper  | JVM did not exit on request, terminated wrapper  |=20
> > Launching a JVM...
> > jvm 2    | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
> > jvm 2    |   Copyright 1999-2006 Tanuki Software, Inc.  All Rights
> > Reserved.
> > jvm 2    |
> >
> is the below error thrown when a restart was attempted by you after=20
> the process was killed with OOM?
>
> > jvm 2    | [15:54:56] ERROR
> > [org.apache.directory.server.core.authz.GroupCache]
> > - Exception while initializing the groupCache:  {}
> > jvm 2    |
> > org.apache.directory.api.ldap.model.exception.LdapOperationErrorExcepti=
on
> > jvm 2    |      at
> >
> org.apache.directory.server.core.partition.impl.btree.AbstractBTreePar
> tition.fetch(AbstractBTreePartition.java:1120)
> > jvm 2    |      at
> >
> org.apache.directory.server.xdbm.search.evaluator.EqualityEvaluator.ev
> aluate(EqualityEvaluator.java:97)
> > jvm 2    |      at
> >
> org.apache.directory.server.xdbm.search.evaluator.AndEvaluator.evaluat
> e(AndEvaluator.java:110)
> > jvm 2    |      at
> >
> org.apache.directory.server.core.partition.impl.btree.EntryCursorAdapt
> or.get(EntryCursorAdaptor.java:167)
> > jvm 2    |      at
> >
> org.apache.directory.server.core.partition.impl.btree.EntryCursorAdapt
> or.get(EntryCursorAdaptor.java:1)
> > jvm 2    |      at
> >
> org.apache.directory.server.core.api.filtering.BaseEntryFilteringCurso
> r.next(BaseEntryFilteringCursor.java:377)
> > jvm 2    |      at
> >
> org.apache.directory.server.core.authz.GroupCache.initialize(GroupCach
> e.java:164)
> > jvm 2    |      at
> >
> org.apache.directory.server.core.authz.GroupCache.<init>(GroupCache.ja
> va:122)
> > jvm 2    |      at
> >
> org.apache.directory.server.core.authz.AciAuthorizationInterceptor.ini
> t(AciAuthorizationInterceptor.java:286)
> > jvm 2    |      at
> >
> org.apache.directory.server.core.DefaultDirectoryService.initIntercept
> ors(DefaultDirectoryService.java:688)
> > jvm 2    |      at
> >
> org.apache.directory.server.core.DefaultDirectoryService.initialize(De
> faultDirectoryService.java:1836)
> > jvm 2    |      at
> >
> org.apache.directory.server.core.DefaultDirectoryService.startup(Defau
> ltDirectoryService.java:1247)
> > jvm 2    |      at
> >
> org.apache.directory.server.ApacheDsService.initDirectoryService(Apach
> eDsService.java:323)
> > jvm 2    |      at
> >
> org.apache.directory.server.ApacheDsService.start(ApacheDsService.java
> :182)
> > jvm 2    |      at
> >
> org.apache.directory.server.wrapper.ApacheDsTanukiWrapper.start(Apache
> DsTanukiWrapper.java:72)
> > jvm 2    |      at
> >
> org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:2
> 788)
> > jvm 2    | Caused by:
> > org.apache.directory.api.ldap.model.exception.LdapOperationErrorExcepti=
on
> > jvm 2    |      at
> >
> org.apache.directory.server.core.partition.impl.btree.AbstractBTreePar
> tition.fetch(AbstractBTreePartition.java:1196)
> > jvm 2    |      at
> >
> org.apache.directory.server.core.partition.impl.btree.AbstractBTreePar
> tition.fetch(AbstractBTreePartition.java:1116)
> > jvm 2    |      ... 15 more
> >
> >
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>


--
Kiran Ayyagari
http://keydap.com