Author: buildbot Date: Mon Dec 24 23:59:00 2012 New Revision: 843750 Log: Staging update by buildbot for directory Added: websites/staging/directory/trunk/content/apacheds/advanced-ug/1.2-network.html websites/staging/directory/trunk/content/apacheds/advanced-ug/1.3-directory-service.html websites/staging/directory/trunk/content/apacheds/advanced-ug/1.4-interceptors.html websites/staging/directory/trunk/content/apacheds/advanced-ug/1.5-backend.html websites/staging/directory/trunk/content/apacheds/advanced-ug/images/architecture.graphml (with props) websites/staging/directory/trunk/content/apacheds/advanced-ug/images/architecture.png (with props) Modified: websites/staging/directory/trunk/content/ (props changed) websites/staging/directory/trunk/content/apacheds/advanced-ug/1-architecture.html websites/staging/directory/trunk/content/apacheds/advanced-ug/1.1-architecture-overview.html websites/staging/directory/trunk/content/apacheds/advanced-ug/3.2-operations-on-an-administrativepoint.html Propchange: websites/staging/directory/trunk/content/ ------------------------------------------------------------------------------ --- cms:source-revision (original) +++ cms:source-revision Mon Dec 24 23:59:00 2012 @@ -1 +1 @@ -1425518 +1425693 Modified: websites/staging/directory/trunk/content/apacheds/advanced-ug/1-architecture.html ============================================================================== --- websites/staging/directory/trunk/content/apacheds/advanced-ug/1-architecture.html (original) +++ websites/staging/directory/trunk/content/apacheds/advanced-ug/1-architecture.html Mon Dec 24 23:59:00 2012 @@ -131,10 +131,10 @@

Chapter content

Modified: websites/staging/directory/trunk/content/apacheds/advanced-ug/1.1-architecture-overview.html ============================================================================== --- websites/staging/directory/trunk/content/apacheds/advanced-ug/1.1-architecture-overview.html (original) +++ websites/staging/directory/trunk/content/apacheds/advanced-ug/1.1-architecture-overview.html Mon Dec 24 23:59:00 2012 @@ -119,7 +119,7 @@
@@ -127,7 +127,14 @@

1.1 - Architecture Overview

-

Blah...

+

The Apache Directory Server (aka ApacheDS) architecture has many different layers. The following schema expose the most important ones :

+

ApacheDS architecture

+

As we can see, we distinguish four different layers : + The network + The Session + The PartitionNexus + The Backends

+

We will describe in detail those layers in the following chapters.

Added: websites/staging/directory/trunk/content/apacheds/advanced-ug/1.2-network.html ============================================================================== --- websites/staging/directory/trunk/content/apacheds/advanced-ug/1.2-network.html (added) +++ websites/staging/directory/trunk/content/apacheds/advanced-ug/1.2-network.html Mon Dec 24 23:59:00 2012 @@ -0,0 +1,185 @@ + + + + + 1.2 - Newtork Layer — Apache Directory + + + + + + +
+ +
+
+ + + +
+
+ + + + + +

1.2 - Network Layer

+

This layer is the part the user connects to when he wants to obtain some data from the server. This is not a mandatory part ot the server : we don't need to use it when the server is embedded.

+

We offer more than just LDAP protocol, the server also include various protocols : + Kerberos + NTP + DHCP + DNS +* ChangePassword

+

Not all of them are implemented in the current version, but at least the Kerberos server is available. (The other protocols have been developped as a proof of concept : as they are all dpeending on a storage database, we have used the LDAP server as a storage).

+

It's perfectly possible to imagine more protocols being implemented in the near future...

+

Server startup

+

This chapter title is a bit misleading. We don't start a server, we start a DirectoryService, then we start various servers on top of it. The DirectoryService is the part responsible for the management f data (retrieval, storage, etc). All the servers can access this storage if needed.

+

So when the DirectoryService has been started and is operational, we can start the various servers, which will accept incoming requests from remote peers.

+

Transports

+

We allow connection through the definition of transports. A Transport is a TCP or an UDP socket capable of absorbing a request and to send a response. Depending on the type of server, we may declare one or more TCP Transports, or a TCP and a UDP Transports, or an UDP Transport only.

+

Ldap Server

+

The LDAP server needs one or two TCP Transport. We have the standard LDAP port (defaulting to 10389 for ApacheDS, but the well know port is usually 389), and one can also declare the LDAPS port (defaulting to 10636 for ApacheDS, but the well know port is usually 636).

+

+Note that LDAPS is considered as deprecated. +

+

Kerberos Server

+

The Kerberos Server uses one TCP Transport (defaulting to 60088, but the well know port is 88 ) and one UDP _transport (same value for both ports). The idea is that the communication starts on TCP and continues on UDP.

+

ChangePassword Server

+

The ChangePassword Server uses one TCP Transport and one UDP transport, too. The default value is 60464, but the well known port is 464.

+

Http Server

+

We have a HttpServer running too, it's used for management. The declared ports are both TCP port, one is for HTTP and its default value is 8080, the other one is for HTTPS and its default value is 8443.

+ + + + + +
+
+
+ +
+ + \ No newline at end of file Added: websites/staging/directory/trunk/content/apacheds/advanced-ug/1.3-directory-service.html ============================================================================== --- websites/staging/directory/trunk/content/apacheds/advanced-ug/1.3-directory-service.html (added) +++ websites/staging/directory/trunk/content/apacheds/advanced-ug/1.3-directory-service.html Mon Dec 24 23:59:00 2012 @@ -0,0 +1,168 @@ + + + + + 1.3 - DirectoryService — Apache Directory + + + + + + +
+ +
+
+ + + +
+
+ + + + + +

1.3 - DirectoryService

+

The DirectoryService is the core of the server. This is where we process incoming requests and ask the backend for data.

+

It has an entry point, the OperationManager, which is in charge of pushing the requests into the Interceptors chain, and to protect the server against concurrent modifications.

+

Then the request is going through every Interceptor being registred for this operation. When we have gone through all the Interceptors, we have reach the PartitionNexus, which is the connection with the backends.

+

We now just have to determinate which type of Backend we should address, and this is done using the Dn.The request is then transmitted to the Backend, which returns the result.

+

The result bubble up through the Interceptors as we unfold the stack stack, up the the OperationManager and the caller.

+

Environment

+

The DirectoryService knows about its execution environment : it has a schemaManager instance, it knows about the Interceptors chain, it stores a map of all the pending requests (it's necessary as one may abandon some request), it holds the existing Sessions.

+

In other word, the DirectoryService is not only the part of teh server executing the logic, it also holds the current state of every clients.

+ + + + + +
+
+
+ +
+ + \ No newline at end of file Added: websites/staging/directory/trunk/content/apacheds/advanced-ug/1.4-interceptors.html ============================================================================== --- websites/staging/directory/trunk/content/apacheds/advanced-ug/1.4-interceptors.html (added) +++ websites/staging/directory/trunk/content/apacheds/advanced-ug/1.4-interceptors.html Mon Dec 24 23:59:00 2012 @@ -0,0 +1,645 @@ + + + + + 1.4 - Interceptors — Apache Directory + + + + + + +
+ +
+
+ + + +
+
+ + + + + +

1.4 - Interceptors

+

Interceptors are functional layers inside the DirectoryService. Ech one of them are responsible for a specific task. They are ordered, and this order is not to be changed.

+

Some Interceptors can be disabled, some other can be enabled. It's also possible to add some new one.

+

All in all, they will handle operations from a specific functional aspect.

+

Handled operations

+

Each Interceptor handle a subset of the possible operations, among those listed in the following table :

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OperationDescription
AddAdd an entry in the backend
BindBind on the DirectoryService
CompareCompare the elements with the associated entry in the backend
DeleteDelete the entry
getRooDSEGet back the RootDSE entry
hasEntryTells if an entry exists
LookupFetch an entry
ModifyModify an entry
MoveMove an entry
MoveAndRenameMove and rename an entry
RenameRename an entry
SearchSearch for entries
UnbindUnbind from the DirectoryService
+

It is important to understand that each operation wil go through each Interceptor that are declared to handle the operation, down to the backend.

+

Existing interceptors

+

The following interceptors are already present in the server, even if they are not enabled. In this table, we list all the operation each interceptor is handling, and if the interceptor is enabled by default or not :

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
InterceptorEnabledaddbndcmpdelDSEhaslkpmodmovm&rrenseaubd
AciAuthorizationInterceptoryesX-XX-XXXXXXX-
AdministrativePointInterceptoryesX--X---XXXX?-
AuthenticationInterceptoryesXXXXXXXXXXXXX
ChangeLogInterceptoryesX--X---XXXX-
CollectiveAttributeInterceptoryesX-----XX---X-
DefaultAuthorizationInterceptoryes---X--XXXXXX-
EventInterceptoryesX--X---XXXX--
ExceptionInterceptoryesX--X---XXXX--
JournalInterceptoryesX--X---XXXX--
KeyDerivationInterceptornoX------X-----
NormalizationInterceptoryesXXXX-XXXXXXX-
OperationalAttributeInterceptoryesX--X--XXXXXX-
PasswordHashingInterceptornoX------X-----
ReferralInterceptoryesX--X---XXXX--
SchemaInterceptoryesX-X---XX-?XX-
SubentryInterceptoryesX--X--?XXXXX-
TimerInterceptornoXXXXXXXXXXXXX
TriggerInterceptoryesX--X---XXXX--
+

Interceptors order

+

As we already said, the Intecreptors order is significant : why would we proceed an Add operation through all the Interceptors if the user is simply denied the right to add an entry by the AciAuthorizationInterceptor ?

+

Currently, the following order is enforced :

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OrderInterceptor
1NormalizationInterceptor
2AuthenticationInterceptor
3ReferralInterceptor
4AciAuthorizationInterceptor
5DefaultAuthorizationInterceptor
6AdministrativePointInterceptor
7ExceptionInterceptor
8SchemaInterceptor
9OperationalAttributeInterceptor
10SubentryInterceptor
11EventInterceptor
12TriggerInterceptor
13ChangeLogInterceptor
14JournalInterceptor
+

Example

+

Let's consider the search operation. It will be processed successuvly by the following Intecreptors, as it can be deduced by the two previous tables :

+
    +
  • NormalizationInterceptor
  • +
  • AuthenticationInterceptor
  • +
  • AciAuthorizationInterceptor
  • +
  • DefaultAuthorizationInterceptor
  • +
  • SchemaInterceptor
  • +
  • OperationalAttributeInterceptor
  • +
  • SubentryInterceptor
  • +
+

We can do the same exercise for each operation.

+

Processing

+

Basically, an Interceptor receives a request for an operation, do some pre-processing, call the next Interceptor in the chain, do some post-processing, and return a result.

+

Calling the next Interceptor is as simple as calling the next(OperationContext) method, which will compute the right Interceptor.

+

The pre-processing and post-processing are standard Java code, there is nothing special there.

+

Each operation is passed into an instance of a specific OperationContext, which contains all what is needed about the operation and the environement.

+ + + + + +
+
+
+ +
+ + \ No newline at end of file Added: websites/staging/directory/trunk/content/apacheds/advanced-ug/1.5-backend.html ============================================================================== --- websites/staging/directory/trunk/content/apacheds/advanced-ug/1.5-backend.html (added) +++ websites/staging/directory/trunk/content/apacheds/advanced-ug/1.5-backend.html Mon Dec 24 23:59:00 2012 @@ -0,0 +1,205 @@ + + + + + 1.5 - Backend — Apache Directory + + + + + + +
+ +
+
+ + + +
+
+ + + + + +

1.5 - Backend

+

The Backend is the part of the server responsible for storing data in a way we can easily retrieve them. This storage does not necessarily have to be remanent : we can have a in-memory backend.

+

Existing Backends

+

We currently have 3 different backends : + JDBM + LDIF +* In-Memory

+

JDBM Backend

+

The JDBM backend is storing data on disk, using BTrees. It's fast when it comes to retrieve data, slow when you have to add them.

+

In-Memory Backend

+

This Backend loads in memory a full set of entries. ALl of them must be hold by the existig memory, we don't write on disk anything nor we read anything from disk. If the server is stopped, everything is lost.

+

LDIF Backend

+

It come sin two forms : one single file, or many fles (one per entry). It's always backed by a in-memory Backend, otherwise it would not be possible to retrieve the entries.

+

As we depend on a in-memory backend, which handles the indexes, we have to create those index when this Backend is read, which can be a costly operation.

+

Future Backends

+

We intend to add another in-memory backend, based on Mavibot, a MVCC BTREE. The biggest advantage over the other systems is that it's fast, it allows concurrent reads without locks when the other Backend block the reads when some write operation is being processed. Also it saves on disk it contents peridodically, and has a Journal so that we can recover fro a crash.

+

The only drawback is that all the entries and indexes must hold in memory. On the other hand, we don't anymore need a cache.

+

How it works

+

Basically, each Backend instance inherit from the AbstractBTreePartition class. We can see that a Backend must be a Btree.

+

Data are stored into various tables. In fact, we have one table containing all the entries - the MasterTable - and many indexes.

+

MasterTable

+

The MasterTable coantins all the entries, serialized.

+

This table is a BTree, where the key is the entry's UUID, and the value the serialized entry.

+

+Theorically, we could be able to read it, and restore the whole database, indexes included, assuming that we know which index we have to create. Sadly, it's not enough, as the entries are stored without any information about their position in the DIT. +

+

Indexes

+

Each index is also a BTree, with some exceptions : as we may store multi-valued elements, it's perfectly possible that the value will grow up to a point it's extremely costly to store it serialized. For instance, the ObjectClass index may have thousands of entries for the Person key.

+

In this case, we use a sub-btree, which is a BTree (as strange as it sounds, it's an easy way to add a new key without having to rewrite the full value).

+

The key can be a String, or a ParentIdAndRdn.

+

We have 7 system indexes, which are created when the server is started :

+
    +
  • ObjectClass : to easily find any entry associated with a give ObjectClass
  • +
  • EntryUUID : The entry unique ID index
  • +
  • EntryCsn : The Change Sequence Number index
  • +
  • ParentIdAndRdn : A special index containing a RDN and its parent
  • +
  • Presence : An index used when searching for the presence of an attributeType in an entry
  • +
  • Alias : An index used for aliases
  • +
  • OneAlias : An index used for children aliases
  • +
  • SubAlias : An index used of descendant aliases
  • +
+

The user may define many different index, dependening on his needs.

+

The ParentIdAndRdn index

+

This index is special, as it's used to associate an entry to a position in the DIT. Assuming that each entry has a Dn, and that this Dn describes a hierarchie, the ParentIdAndRdn index depicts this hierarchy.

+

The ParentId part refers to the UUID of the parent for the current entry. The Rdn part is the entry Rdn. In order to rebuild the full Dn for a given entry, we must get all the ParentIdAndRdn up to the root to grab all the needed Rdn.

+

This index is also used to process one level and sub level indexes.

+ + + + + +
+
+
+ +
+ + \ No newline at end of file Modified: websites/staging/directory/trunk/content/apacheds/advanced-ug/3.2-operations-on-an-administrativepoint.html ============================================================================== --- websites/staging/directory/trunk/content/apacheds/advanced-ug/3.2-operations-on-an-administrativepoint.html (original) +++ websites/staging/directory/trunk/content/apacheds/advanced-ug/3.2-operations-on-an-administrativepoint.html Mon Dec 24 23:59:00 2012 @@ -128,12 +128,12 @@

Operations

There are six kind of operations we can have on an AdministrativePoint : -- creating a new AP -- removing an existing AP -- modifying an existing AP by adding or removing a role -- renaming an AP -- moving an AP -- renaming or moving an AP

+ creating a new AP + removing an existing AP + modifying an existing AP by adding or removing a role + renaming an AP + moving an AP + renaming or moving an AP

Renaming an AP has no impact on the administrative model, as we don't point (internally) on the entry's DN, but on its UUID, so the last three operations can be gathered into one single Move operation.

Added: websites/staging/directory/trunk/content/apacheds/advanced-ug/images/architecture.graphml ============================================================================== Binary file - no diff available. Propchange: websites/staging/directory/trunk/content/apacheds/advanced-ug/images/architecture.graphml ------------------------------------------------------------------------------ svn:mime-type = application/xml Added: websites/staging/directory/trunk/content/apacheds/advanced-ug/images/architecture.png ============================================================================== Binary file - no diff available. Propchange: websites/staging/directory/trunk/content/apacheds/advanced-ug/images/architecture.png ------------------------------------------------------------------------------ svn:mime-type = image/png