directory-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Directory Wiki] Update of "Asn1Home" by EmmanuelLecharny
Date Sun, 04 Sep 2005 08:01:54 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Directory Wiki" for change notification.

The following page has been changed by EmmanuelLecharny:
http://wiki.apache.org/directory/Asn1Home

The comment on the change is:
Updated the page.

------------------------------------------------------------------------------
  
  == Components ==
  
- Communications between clients and server can be seen as a two ways / multi ''layers'' system.
The client submit a request to the server, which reply.
+ Communications between clients and server can be seen as a two ways / multi ''layers'' system.
The client submits a request to the server, which replies.
  
- ''Layers'' are just used to facilitate the implementation of this communication. From the
developer point of view, working on a specific level, he just has to know the two layers above
and under.
+ ''Layers'' are just used to facilitate the implementation of this communication. From the
developer point of view, working on a specific level, he just has to know the two layers above
and under, but can be seen as a communication at she same level between the client and the
server. Here is a view of these layers :
  
- {{{
+ attachment:cs-stack.png
  
-    CLIENTS        SERVERS
+ We have depicted three layers :
+  * Request/Response : This is the more abstract layer. Exchanged messages are 'human readable'.
Each message is a Java Bean, containing all the information about a Request or a Response.
+  * PDU : As communication petween the Client and the Server is done through a network, we
need to transform the beans to something more 'network friendly'. The data are stored in PDU,
or '''P'''acket '''D'''ata '''U'''nit. Those PDU contain an encoded form of messages, specified
in [http://www.faqs.org/rfcs/rfc2251.html RFC 2251] and [http://asn1.elibel.tm.fr/en/standards/index.htm
ASN.1]
+  * [http://java.sun.com/j2se/1.4.2/docs/api/java/nio/ByteBuffer.html ByteBuffers] : To transfer
PDU from/to Client to/from Server, we need to store bytes in a structure that will permit
to deal with network latency. Thus we are using byte buffers, which allow us to send pieces
of PDU until the whole PDU has been transmitted. (Note : [http://java.sun.com/j2se/1.4.2/docs/api/java/nio/ByteBuffer.html
ByteBuffer] is also a Java NIO class, but can be seen just as a byte container. It could have
been something totally different from the NIO class).
  
-     STUBS <......> STUBS
-      | ^            | ^
-      V |            | |
-     TLVs <........> TLVs
-      | ^            | ^
-      V |            | |
-    Buffers <....> Buffers
-      | ^            | ^
-      V |            | |
-      IO <.........> IO
-      | ^            | ^
-      | |            | |
-      | +------------+ |
-      +----------------+
- }}}
- 
- It allows many different implementations, as of :
+ This layering allows many different implementations. 
-  * smart stubs, that are able to generate directs IO;
-  * dumb stubs, that communicate with dumb Tuples, ...;
-  * semi-smart Stubs, that communicate with TLVs, avoiding the Tuples layer
-  * pre-built elements for standards queries or replies (errors, ...)
-  * semi-built elements that just need a single modification (Message Id, ...)
  
  One can also imagine inter-layers used to trace debug informations.
  
- Debugging is easier : you can analyze the upper level, then the lower one, and so on.
+ Inter layer communication rely on a pipe-line : each layer push some piece of information
to the next layer (up or down), and so on.
  
- Inter layer communication rely on a pipe-line : each layer push some piece of information
to the net layer (up or down), which will do the same.
- 
- Each layer may implement its own strategy to fulfill the reception and transmission of this
data :
+ Each layer may implement its own strategy to fulfill the reception and transmission of the
data it is responsible of :
   * emission
    * asynchronous push
    * synchronous push
@@ -54, +34 @@

    * established and dedicated channel
    * multiplexed channel
  
+ === POJOs ===
+ '''POJOs''' are Java classes that contain high level informations.
  
- 
- === Stubs/POJOs ===
- '''Stubs''' or '''POJOs''' are Java classes that contain high level informations.
- 
- A client create an ''instance'' of a stub to communicate with the server, which create an
other one to reply. They implement a kind of ''application layer'' between clients and server.
+ A client create an ''instance'' of a class to communicate with the server, which create
an other one to reply. They implement a kind of ''application layer'' between clients and
server.
- 
  
  Ideally, they are generated by an '''ASN.1''' compiler, but can be hand crafted.
  
- === PDU and TLVs ===
+ === PDUs/TLVs ===
  
  PDU stands for '''P'''acket '''D'''ata '''U'''nit. An ASN.1 encoded element is stored in
a PDU. This is what is transfered between a client and a server.
  
@@ -73, +50 @@

  Further information about '''TLV'''s can be found here :
   * ["TLVPageInfo"]: Informations about '''Tlv'''s
  
- === Buffer ===
+ === ByteBuffer ===
  Buffering the incoming request or the ourgoing response is essential. As a request or a
response can be hudge (for exampole, if we want to store images), it is necessary to store
bytes in buffers in order to be able to pipeline the processing. Flushing informations byte
by byte is totally insane, from the network point of view.
  
- We are using the '''NIO''' '''ByteBuffer''' structure to store chunks of information, before
pushing them on the network, and reversly, store incoming bytes into buffers before processing
the request.
+ We are using the '''NIO''' [http://java.sun.com/j2se/1.4.2/docs/api/java/nio/ByteBuffer.html
ByteBuffer] structure to store chunks of information, before pushing them on the network,
and reversly, store incoming bytes into buffers before processing the request.
- 
- === IO ===
- For any information, the best place is certainly ["MinaHome"]
  
  == Compiler ==
  
@@ -90, +64 @@

  There are two kind of processing : '''encoding''' and '''decoding'''. Encoding is quite
easy, decoding is much more complicated.
  
  {{{
- Important : decoding an ASN.1 PDU is generally not possible if you have no knowledge of
the grammar being decoded. To limit the size of PDUs, the encoding schemes used (PER, DER,
BER, ...) permit the elimination of some TL if the constructed TLV that encapsulate the previous
one is unambiguiously known. One who want to decode a PDU *MUST* know which grammar has been
used.
+ Important : decoding an ASN.1 PDU is generally not possible if you have no knowledge of
the grammar being decoded. To limit the size of PDUs, the encoding schemes used (PER, DER,
BER, ...) permits the elimination of some TL if the constructed TLV that encapsulate the previous
one is unambiguiously known. One who want to decode a PDU *MUST* know which grammar has been
used.
  }}}
  
  === Encoder ===
  
- The encoding process is quite easy. As we know what has to be encoded, the structure of
the PDU is somehow dependent on the structure of the POJO which contains the data. The only
tricky things is the '''Length''' part, which has to be computed. As a '''TLV''' may have
a '''V''' part which is itself one or more '''TLV'''s, its '''L''' part will be the sum of
each '''TLV'''s length. This is typically a recursive processing, but we can also process
the POJO in two passes :
+ The encoding process is quite easy. As we know what has to be encoded, the structure of
the PDU is somehow dependent on the structure of the POJO which contains the data. The only
tricky things is the '''Length''' part, which has to be computed. As a '''TLV''' may have
a '''V''' part which is itself one or more '''TLV'''s, its '''L''' part will be the sum of
each included '''TLV'''s length. This is typically a recursive processing, but we can also
process the POJO in two passes :
   * the first pass compute each length
   * the second pass generate the '''PDU'''
  
@@ -108, +82 @@

  
  === Performance ===
  TODO : performance against memory/scalability/failover
- TODO : which kind of performance should we deliver? Maximum throughput = bandwith/average
PDU size. For instance, with a 1Gb network connection, assuming that we have an average PDU
size of 100 bytes, the system should deliver 1 M Pdu/s, not to mention TCP/IP overloading
(typically, a TCP/IP packet size is 1500 bytes, so the previous number is to be divided by
15 : 66 000 PDU/s).
+ TODO : which kind of performance should we deliver? Maximum throughput = bandwith/average
PDU size. For instance, with a 1Gb network connection, assuming that we have an average PDU
size of 100 bytes, the system must deliver 1 M Pdu/s to saturate the network.
  
- Actually, the new decoder eats 130 000 BindRequest PDU per second, but we have to take into
account the works that must be done aside.
+ Actually, the new decoder eats 110 000 BindRequest PDU or 37 000 SearchResultEntry PDU per
second on my 2.8Ghz computer, but we have to take into account the works that must be done
aside.
- 
- So, ideally, treating a PDU in 15 ns might be enough.
- 
- === Memory consumption ===
- 
- Memory is limited by the '''JVM''' parametrization. the '''CODEC''' system should never
cause a memory allocation. Clients side is not a problem, but server side is. We '''must'''
build a mechanism that works with a fixed amount of memory. In case of memory lack, the system
'''may''' discards some entering requests, or discard greedy requests, or flush to the disk
those greedy request for a postponed treatment.
- 
- This mechanism '''should''' be light, to obtain an acceptable level of performance.
- 
- A possible mechanism could be to allocate some instances, and associates them to a thread,
which will be in charge of the codec processing.
- 
- A server will contains a limited number of codec threads, each one with its codec instances
pool (no synchronisation)
- 
- A global pool of instance could be allocated on initialization, depending on the global
memory. each thread will ask for an instance and will keep it in its own pool.
- 
- The new implementation create a pool per object's type for each decoding thread, and a global
synchronized pool. When a thread need a new object, either it find it in its local pool, or
it asks for a new one from the global pool.
- 
- To limit the memory consumption, we must also implement a requisition mechanism where the
exhausted global pool asks local pools some free objects. When no more objects are available
on local pools, a choice must be made to kill or serialize a decoder, freing the associated
objects. This mechanism is still to be implemented.
- 
- Another problem is the String allocations. Strings are immutable, so they can't be reused.
It's not possible to create a String pool, so memory is not manageable with Strings. We have
two ways here :
-  - create a new MutableString class, which is quite a piece of work, because we need to
handle UTF-8
- 
-  - pre-allocate as much as String than we can, up to the StringPool limit, and each time
we need to create a new String, we will just have to nullify the pre-allocated one. As String
could have different sizes, we have to pre-allocate Strings with different sizes, like 16-chars
length Strings, 32-chars Strings, and so on. If we have to allocate, say, a 44 char long String,
we will free a 48-chars String, so the memory will nevcer be exhausted. If there is no more
48-chars String available, then we could free either a 64-chars String (or a bigger one),
or free one 32-chars String and a 16-chars String. Strings above 1024 bytes will be streamed.
- 
- This may seems very complicated, but we have to deal with this kind of constraints, as the
PDU that we can receive may be really strange, or built to break the server. 
- 
- 
  
  == Implementations ==
  Two codecs are currently available :

Mime
View raw message