syncope-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francesco Chicchiriccò (Jira) <j...@apache.org>
Subject [jira] [Resolved] (SYNCOPE-1485) Reindex of elasticsearch ends with memory error in case of huge amount of data
Date Wed, 04 Sep 2019 12:36:00 GMT

     [ https://issues.apache.org/jira/browse/SYNCOPE-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Francesco Chicchiriccò resolved SYNCOPE-1485.
---------------------------------------------
    Resolution: Fixed

> Reindex of elasticsearch ends with memory error in case of huge amount of data
> ------------------------------------------------------------------------------
>
>                 Key: SYNCOPE-1485
>                 URL: https://issues.apache.org/jira/browse/SYNCOPE-1485
>             Project: Syncope
>          Issue Type: Bug
>          Components: core, extensions
>    Affects Versions: 2.0.13, 2.1.4
>            Reporter: fabio martelli
>            Assignee: Francesco Chicchiriccò
>            Priority: Major
>             Fix For: 2.0.14, 2.1.5, 3.0.0
>
>
> In case of a huge amount of data, elastic search reindex ends with memory error.
> See below an extract of the stacktrace.
> {code}
> 12:28:21.014 ERROR org.apache.syncope.core.provisioning.api.job.SchedTaskJobDelegate
- While executing task 0dfe540f-67aa-41a1-be54-0f67aa71a1f7
>  org.quartz.JobExecutionException: While rebuilding index for domain Master
>  at org.apache.syncope.core.provisioning.java.job.ElasticsearchReindex.doExecute(ElasticsearchReindex.java:124)
~[syncope-ext-elasticsearch-provisioning-java-2.1.4.jar:2.1.4]
>  at org.apache.syncope.core.provisioning.java.job.AbstractSchedTaskJobDelegate.execute(AbstractSchedTaskJobDelegate.java:122)
~[syncope-core-provisioning-java-2.1.4.jar:2.1.4]
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212]
>  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212]
>  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_212]
>  at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
>  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
~[spring-aop-5.1.6.RELEASE.jar:5.1.6.RELEASE]
>  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
~[spring-aop-5.1.6.RELEASE.jar:5.1.6.RELEASE]
>  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
~[spring-aop-5.1.6.RELEASE.jar:5.1.6.RELEASE]
>  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
~[spring-tx-5.1.6.RELEASE.jar:5.1.6.RELEASE]
>  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
~[spring-tx-5.1.6.RELEASE.jar:5.1.6.RELEASE]
>  at org.apache.syncope.core.persistence.jpa.spring.DomainTransactionInterceptor.invoke(DomainTransactionInterceptor.java:60)
~[syncope-core-persistence-jpa-2.1.4.jar:2.1.4]
>  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
~[spring-aop-5.1.6.RELEASE.jar:5.1.6.RELEASE]
>  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
~[spring-aop-5.1.6.RELEASE.jar:5.1.6.RELEASE]
>  at com.sun.proxy.$Proxy355.execute(Unknown Source) ~[?:?]
>  at org.apache.syncope.core.provisioning.java.job.TaskJob.lambda$execute$0(TaskJob.java:87)
~[syncope-core-provisioning-java-2.1.4.jar:2.1.4]
>  at org.apache.syncope.core.spring.security.AuthContextUtils.execWithAuthContext(AuthContextUtils.java:117)
~[syncope-core-spring-2.1.4.jar:2.1.4]
>  at org.apache.syncope.core.provisioning.java.job.TaskJob.execute(TaskJob.java:76) ~[syncope-core-provisioning-java-2.1.4.jar:2.1.4]
>  at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz-2.3.1.jar:?]
>  at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) ~[quartz-2.3.1.jar:?]
>  Caused by: org.apache.openjpa.persistence.PersistenceException: Ran out of memory retrieving
query results. {prepstmnt 1711473022 SELECT t0.id, t2.id, t2.ACCOUNTPOLICY_ID, t3.id, t4.id,
t4.ACCOUNTPOLICY_ID, t4.nam
>  e, t4.PARENT_ID, t4.PASSWORDPOLICY_ID, t3.bundleName, t3.connRequestTimeout, t3.connectorName,
t3.displayName, t3.jsonConf, t3.location, t3.maxIdle, t3.maxObjects, t3.maxWait, t3.minEvictableIdleTimeMillis,
t3.min
>  Idle, t3.version, t2.createTraceLevel, t2.deleteTraceLevel, t2.enforceMandatoryCondition,
t2.jsonConf, t2.overrideCapabilities, t2.PASSWORDPOLICY_ID, t2.propagationPriority, t2.provisioningTraceLevel,
t2.PULLPOLIC
>  Y_ID, t2.PUSHPOLICY_ID, t2.randomPwdIfNotProvided, t2.updateTraceLevel FROM SyncopeUser
t0 INNER JOIN SyncopeUser_ExternalResource t1 ON t0.id = t1.user_id INNER JOIN ExternalResource
t2 ON t1.resource_id = t2.id 
>  LEFT OUTER JOIN ConnInstance t3 ON t2.CONNECTOR_ID = t3.id LEFT OUTER JOIN Realm t4
ON t3.ADMINREALM_ID = t4.id ORDER BY t0.id ASC} [code=0, state=53200]
>  at org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:5250) ~[openjpa-jdbc-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(DBDictionary.java:5210)
~[openjpa-jdbc-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:134) ~[openjpa-jdbc-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:115) ~[openjpa-jdbc-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:67) ~[openjpa-jdbc-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.handleCheckedException(SelectResultObjectProvider.java:162)
~[openjpa-jdbc-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.datacache.QueryCacheStoreQuery$CachingResultObjectProvider.handleCheckedException(QueryCacheStoreQuery.java:775)
~[openjpa-kernel-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.kernel.QueryImpl$PackingResultObjectProvider.handleCheckedException(QueryImpl.java:2231)
~[openjpa-kernel-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.lib.rop.EagerResultList.<init(EagerResultList.java:42) ~[openjpa-lib-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1311) ~[openjpa-kernel-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1062) ~[openjpa-kernel-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:912) ~[openjpa-kernel-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:843) ~[openjpa-kernel-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:601) ~[openjpa-kernel-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:297) ~[openjpa-persistence-3.1.0.jar:3.1.0]
>  at org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:314) ~[openjpa-persistence-3.1.0.jar:3.1.0]
>  at org.apache.syncope.core.persistence.jpa.dao.JPAUserDAO.findAll(JPAUserDAO.java:255)
~[syncope-core-persistence-jpa-2.1.4.jar:2.1.4]
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212]
>  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212]
>  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_212]
>  at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
>  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
~[spring-aop-5.1.6.RELEASE.jar:5.1.6.RELEASE]
>  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:206)
~[spring-aop-5.1.6.RELEASE.jar:5.1.6.RELEASE]
>  at com.sun.proxy.$Proxy86.findAll(Unknown Source) ~[?:?]
>  at org.apache.syncope.core.provisioning.java.job.ElasticsearchReindex.doExecute(ElasticsearchReindex.java:85)
~[syncope-ext-elasticsearch-provisioning-java-2.1.4.jar:2.1.4]
>  ... 19 more
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Mime
View raw message