incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Karasulu" <aok...@bellsouth.net>
Subject RE: Official Apache Directory Project Proposal Submission
Date Fri, 12 Sep 2003 02:07:14 GMT
Wow you guys are getting real deep.  Perhaps we should take these technical
conversations over to the ldapd dev list for now.  This place is for
incubator stuff not tech stuff so we'll continue at ldapd-devel after this
bridge email.

BTW I think BerkeleyDB was faster than the jdbc based implementation but not
jump out at you faster like we thought.  The backend design using bdb is
pretty much the same as jdbm.  The difference was the JNI performance
degradation - yes all that copying from crossing the java/c barrier back and
forth slowed us down.  Bdb is great for C but a pure Java implementation
like Jdbm is best.

Once we tried Jdbm we were very happy with the results.  Performance went
through the roof.  And btw an RDBMS is a hog for an LDAP server backing
store.  All the SQL overhead makes it so.  I have to agree with the OpenLDAP
folks - they know what they're talking about.

Alex

-----Original Message-----
From: Robb Penoyer [mailto:rpenoyer@timgia.com] 
Sent: Thursday, September 11, 2003 8:57 PM
To: general@incubator.apache.org
Subject: Re: Official Apache Directory Project Proposal Submission


Hi Jim,

The original pre-release versions of LDAPd were implemented with a 
BerkeleyDB backend, with custom index management etc, much like the 
openldap articale you reference. Those early designs, did have a contracted 
backend store interface defined (thank-you Mr magic - Alex), and indeed 
there is a basic SQL backend implementation in alpha 0.7  Taking into 
perspective the second reference to IBM you provided. We measured the 
performance of a very strongly tuned Oracle database against the BerkleyDB 
implementation and found virtually no performance difference. Albeit, these 
were not formal tests, but they were exactly the same. (hardware, test 
cases etc).

We moved to a default backend based upon the JDBM implementation Alex 
referenced earlier. The performance improvement was staggering, to say the 
least. The nature of LDAP is for high performance read operations, a pure 
indexing mechanism turned out to be ideal. With JDBM everything amounts to 
an index in relational terms. This would absolutely become a problem for a 
standard transactional application, primarily because the biggest struggle 
for our JDBM implementation is how to handle duplicate entries - it costs 
us performance.

Here is where I see it fall out:
   RDBMS : hard to beat for pure transactional power. If you have to store, 
retrieve, update and delete, all the operations will generally work within 
the same performance envelope.

   OODBMS: hard to beat for pure synergy between logic layer and storage, I 
admit a personal weakness with these types of databases.

   Heriarchical Databases: great for maintaining a complete picture of 
complex models (many to many parent child relationships) This is likely 
where stored procedures prove important in RDBMS technology (as you pointed 
out).

I'll say it again, LDAP is NOT a database, it just needs one. That's what 
IBM was saying. the openldap folks retrofitted BerkeleyDB. We went a 
different way. The only real distinction on this front, is that we 
recognixed very early, that the nature of the backend store will dictate 
the performance of your LDAP server "in the context it is designed for". 
Meaning, it is likely that an RDBMS backend associated with LDAP will give 
a better overall performance for a heavily modified directory information 
tree, but will never outperform the raw search capabilities of a purely 
indexed backend for searches. So we leave it up to the implementors. At one 
time we spoke about actually measuring all this stuff and providing 
guidelines - it's something we would love to get work going on --- hint.

Turning over to how LDAP could impact database technologies - let's bring 
it up a layer above the data store. A solid LDAP implementation to protocol 
compliance requires some truly industrial strength mechanisms (beyond data 
storage): schema management, access controls, protocol encoding/decoding, 
search optimizers, providers and on and on and on. If this sounds similar 
to a database implementation, stop wondering. The core of an LDAP server 
performs the same basic functions as a database management system.

What if you added the missing pieces, for example a SQL parser, a JDBC 
driver, a transaction manager, stored procs - but kept the LDAP protocol 
requirements of  forwarding and authoritative areas. You are now in a 
scenario where one LDAP server is representing a database. What kind of 
database, the one you chose, Berkeley, Oracle, SQL Server (yuck), DB2, JDBM.

Now add more LDAP protocol stuff, replication and referrals. You can now 
had a set of LDAP servers acting in concert as 1 database. What storage 
mechanism, how about one RDBMS, one JDBM, and one oodbms. Each configured 
with a schema designed specifically to take advantage of their performance 
characteristics. You now have the best of all worlds - one interface. 
pretty cool huh?

Robb


At 01:49 AM 9/12/2003 +0200, you wrote:
>Alex and Brian,
>
>Regarding the relationship between RDBMS and LDAP...
>
>I believe this document says why RDBMS is wrong for LDAP:
>
>http://www.openldap.org/faq/data/cache/378.html
>
>On the other hand IBM have implemented LDAP in DB2. See:
>
>http://www.research.ibm.com/journal/sj/392/shi.html
>
>Since reading that I have got quite carried away designing
>and implementing heirarchical structures in RDBMS's.
>Mainly designing actually, but the demo referenced from
>my signature below implements heirarchical categories
>of contacts using the same principles.
>
>I understand this project is not just about the protocol but
>about the directory. It seems to me that it is very valuable
>to have a single DBMS that supports both relational and
>heirarchical structures as efficiently as possible. (In fact
>I would suggest not just heirarchies but directed graphs.
>I.e. a child can have one or more parents.)
>
>If the IBM way (that I adapted) turns out to be one of the
>best (following project design) then one thing that is important
>is that you can efficiently add, remove and traverse nodes
>in a tree represented by lots of small RDB records.
>This becomes important for deep heirarchies. I guess
>stored procedures might help in an standard RDBMS.
>
>I might be interested in getting involved.
>
>Regards,
>
>Jim Wright
>
>--
>Recently completed - Child Brain Injury Trust Admin System
>http://cbitdemo.paneris.org/
>
>Urgently seeking paid work
>Java, Linux, XML and much more.
>http://be.webz.cz/
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>For additional commands, e-mail: general-help@incubator.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Mime
View raw message