syncope-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francesco Chicchiriccò <>
Subject Re: [DISCUSS] Manage millions of identities
Date Mon, 29 Oct 2018 10:27:23 GMT
Hi Guido,

On 24/10/18 20:51, Guido Wimmel wrote:
> Hi,
> Am 16.10.18 um 10:54 schrieb Francesco Chicchiriccò:
>> Hi all,
>> I think it's time to discuss about how we want to get prepared for 
>> scenarios where the number of identities (mostly users, for the vast 
>> majority) to manage is considerably high - from 1 million to above; 
>> the typical case being CIAM (Customer IAM).
>> In the IdM deliveries I've been involved so far, scaling Apache 
>> Syncope up to hundreds of thousands of identities is not trivial, but 
>> doable: naturally, most of optimization work shall be done at DBMS 
>> level, as that is obviously the component which is stressed more.
>> I think we can agree about the fact that, in such scenarios, the most 
>> critical data are the ones bound to the actual identities (hence no 
>> connectors, resources, tasks, reports or any other configuration): 
>> consider that with 1 million users and 10 attributes for each user, 
>> we have the following table sizing to deal with:
>> * SyncopeUser: 1M rows
>> * UPlainAttr: 10M rows
>> * UPlainAttrValue: 10M rows
>> Moreover, the search views [1] are all on the same size order 
>> (although one can enable the Elasticsearch extension in such cases, 
>> to improve performances).
>> I think this is what we need to change in order to get better results.
>> So far, I have been able to think of a couple of possibilities:
>> 1. Leverage the JSON column support provided by PostgreSQL [2], MySQL 
>> [3], SQL Server [4] and Oracle DB [5] to extend the current 
>> OpenJPA-based persistence layer
>> Pros:
>>  * reduce the sizing problems by removing the need of UPlainAttr and 
>> UPlainAttrValue tables, search views and joins
>>  * limited implementation effort, as  most of the current JPA layer 
>> can be retained
>>  * keep enjoying the benefits of referential integrity and other 
>> constraints enforced by DBMS (including UNIQUE)
>> Cons:
>>  * each DBMS provides JSON support in its own fashion: implementation 
>> wouldn't be trivial (while we can make it incremental, and add 
>> support for one DBMS at a time)
>>  * scaling capabilities and performance might be overrated - even 
>> though there seems to be very nice references, at least for 
>> PostgreSQL [6][7]
>> 2. Implement a new persistence layer based on a different technology 
>> - I have done some experiments with Apache Cassandra [8] and the 
>> Datastax Java Driver [9]
>> Pros:
>>  * built native for scalability and high availability
>>  * proven and widespread adoption
>>  * Object Mapper [10] allows to semi-transparently convert between 
>> query results and domain, somehow similar as JPA's EntityManager
>> Cons:
>>  * relations are obviously not available, only custom types [11]: the 
>> persistence model shall be redesigned to cope with such situation
>>  * constraints are not available - more specifically UNIQUE, which 
>> will require additional code handling
>>  * implementation effort: all the persistence layer shall be redone, 
>> not only identity-related entities as User, UPlainAttr, 
>> UPlainAttrValue...
>> Besides the two above, there are of course other options in the NoSQL 
>> world (Neo4j, MongoDB, ...), but I am afraid they all present similar 
>> challenges as Cassandra.
>> WDYT?
>> Regards.
> I'd expect it should be possible to make the current relational model 
> work for hundreds of thousands - millions of identities. This should 
> not be too much data for enterprise-grade databases like Oracle or 
> PostgreSQL.
> We have a deployment with approx. a million identities (however, we 
> mostly use basic features of Syncope, and had to do some tweaking on 
> the search queries).
> Maybe one could document the required optimizations / partially 
> integrate them into Syncope? (possibly additional indexes / optimized 
> queries / ...)

First of all, this is an interesting confirmation: (1) the current model 
can handle (in your experience) "hundreds of thousands - millions of 
identities" and (2) you have a deployment with approx. a million identities.

Documenting the required optimizations, or integrating something into 
Syncope is definitely worthwhile: could you share these somehow, even as 
descriptions into an improvement on JIRA?

> For even larger numbers, I'd find both suggestions interesting. I 
> think one would have to do spikes in order to evaluate the performance 
> gain for large numbers of identities for different functionalities. 
> Maybe one could even support both, so that users could choose 
> according to their requirements / risk tolerance.

I am currently in the middle of a spike which leverages PostgreSQL's 
JSONB data type to replace *PlainAttr / * PlainAttrValue, and I am 
around 90% feature-wise.
After that, I would also like to add a new module to the sources, with 
purpose of running performance tests with JMeter support: in this way we 
will be able to effectively check the numbers of the available 

Anyway, given the way how the code is structured from Syncope 2.0 
onwards, we are simply providing different implementations of the 
interfaces in syncope-core-persistence-api:

* syncope-core-persistence-jpa is the current implementation
* syncope-core-persistence-jpa-pgjsonb could be the name of the one I am 
working on (which is actually an extension of the former)
* syncope-core-persistence-jpa-mysqljson could be the same approach as 
above, but for MySQL's JSON data type
* syncope-core-persistence-cassandra, syncope-core-persistence-mongodb, 
syncope-core-persistence-whatever could be provided at any point in time

This to confirm that, in response to your suggestion and to concerns 
raised in other e-mails of this thread, there is always room to provide 
new implementations to support virtually any persistence technology; and 
that users will be free to choose one based on their needs by simply 
selecting the correct Maven dependency to include in their own projects.


>> [1] 
>> [2]
>> [3]
>> [4] 
>> [5]
>> [6] 
>> [7] 
>> [8]
>> [9]
>> [10] 
>> [11] 

Francesco Chicchiriccò

Tirasa - Open Source Excellence

Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail

View raw message