directory-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From akaras...@apache.org
Subject svn commit: r154725 [2/2] - in incubator/directory/asn1/branches/rewrite: ber/xdocs/ codec/xdocs/ stub-compiler/xdocs/
Date Mon, 21 Feb 2005 21:44:49 GMT
Added: incubator/directory/asn1/branches/rewrite/ber/xdocs/eureka.xml
URL: http://svn.apache.org/viewcvs/incubator/directory/asn1/branches/rewrite/ber/xdocs/eureka.xml?view=auto&rev=154725
==============================================================================
--- incubator/directory/asn1/branches/rewrite/ber/xdocs/eureka.xml (added)
+++ incubator/directory/asn1/branches/rewrite/ber/xdocs/eureka.xml Mon Feb 21 13:44:46 2005
@@ -0,0 +1,929 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<document>
+  <properties>
+    <author email="akarasulu@apache.org">Alex Karasulu</author>
+    <title>Eureka!</title>
+  </properties> 
+  
+ <body>
+
+  <section name="And we saw the light ... ">
+     <p>
+       One day Wes and Alex started talking about going to town on a new 
+       ASN.1 BER Library and here's what happened ...
+     </p>    
+     <subsection name="The Conversation">
+      <source>
+[SNIP]
+
+Wes says:
+I've been thinking about the decoding process a bit over the weekend.
+
+Alex Karasulu says:
+k I'm listening
+
+Wes says:
+and encoding.
+
+Wes says:
+I'm not sure at the initial stage there will be *one* decoder.
+
+Wes says:
+We will need some place to hold our TLV tree.
+
+Wes says:
+and also, I was thinking about really long messages.
+
+Alex Karasulu says:
+you need multiple codecs (coder decoders)
+
+Alex Karasulu says:
+right
+
+[SNIP]
+
+Wes says:
+We got one part that builds the tree
+
+Wes says:
+part two should be the translation.
+
+[SNIP]
+
+Wes says:
+I think the only issue we have is how to handle chunking, and blocking versus 
+non-blocking code.
+
+Wes says:
+And also, dealing with really huge messages.
+
+Wes says:
+It obviously won't make sense to build a TLV tree in its entirety for a huge 
+search result.
+
+Alex Karasulu says:
+right I agree
+
+Alex Karasulu says:
+for encoding there is a mechanism for breaking down large TLVs of simple types 
+down 
+
+Wes says:
+encoding is a non issue as far as chunking goes.
+
+Alex Karasulu says:
+basically in the book they talk about 3 ways of specifying length
+
+Alex Karasulu says:
+the L part
+
+Alex Karasulu says:
+right but it effects decoding
+
+Alex Karasulu says:
+but if another provider is doing encoding I see what you mean
+
+Alex Karasulu says:
+basically we can break stuff down by injecting the 3rd indeterminate length form
+
+Alex Karasulu says:
+follow me out
+
+Wes says:
+You give the encoder an output interface, and every time it fills up the byte 
+bufffer, it spits it out.
+
+Alex Karasulu says:
+Strictly talking about decoding and chunk sizes for now.
+
+Wes says:
+K.  decoding then.
+
+Alex Karasulu says:
+just for background - you read the section on the 3 different modes for 
+specifying length right: short, long and indeterminant?
+
+[SNIP]
+
+Alex Karasulu says:
+Your reading and encounter a really big simple type using the long encoding for 
+L.  So you know what you have to read is a hugh blob of data in one big hunk.  
+Basically there is some threashold u use to judge whether or not the blob is too
+big and needs to be chopped up.
+
+Wes says:
+I did read the section the length.
+
+Alex Karasulu says:
+cool
+
+Wes says:
+I actually printed the whole appendix out and read it.
+
+Wes says:
+on BER.
+
+Alex Karasulu says:
+cool that's what I was reffering to
+
+Wes says:
+An encoder can choose any one he wants.
+
+Alex Karasulu says:
+Now your decoder can break down the long format into the indeterminate format 
+nesting smaller TLVs inside the TLV.  Hence converting the simple TLV into a 
+constructed one.
+
+Alex Karasulu says:
+The key here is not to keep all the tlvs in memory or the entire encoded buffer 
+in memory
+
+Wes says:
+For decoding, there are messages where keeping the intermediate form in memory 
+is not an issue, and with others, there are.
+
+Wes says:
+issues.
+
+Alex Karasulu says:
+Right depends on the message size
+
+Wes says:
+The client will want to process most of the messages as a complete object.
+
+Wes says:
+By definition, it will be in memory.
+
+Alex Karasulu says:
+Yeah I know what you are saying.  We need to make the library not do this 
+though.  Then there would be more than one copy in memory.  Leave it upto the 
+user to determine how the data is dealt with.  Eventually we can take messures 
+to stream data if we want instead of having it all in memory.
+
+Wes says:
+Back up just a second.
+
+Alex Karasulu says:
+There are funky tactics we can employ way down the road - but for the time 
+being lets make it so our codecs dont need massive footprints 
+
+Alex Karasulu says:
+sure talk to me
+
+Wes says:
+I used this technique in a Btrieve interface I wrote for U. S. South...
+
+Wes says:
+which I stole from OpenTDS.
+
+Alex Karasulu says:
+Btrieve?
+
+Wes says:
+Yea, an ISAM database.
+
+Alex Karasulu says:
+oh ok
+
+Wes says:
+It used byte buffers to send and retrieve records.
+
+Wes says:
+I wrote a java class that basically treated the byte array as primitives.
+
+Alex Karasulu says:
+cool so you're already of the mindset to keeping the decoding and encoding 
+memory footprints small
+
+Wes says:
+That might not work with us though.
+
+Wes says:
+It might.
+
+Wes says:
+All we need to know
+
+Wes says:
+is that this field goes with this TLV.
+
+Wes says:
+and convert it on the fly.
+
+Wes says:
+Also, we an simply dump the TLVs when we are done.
+
+Alex Karasulu says:
+yeah that's part of some tables we may need to maintain with a mappiung
+
+Alex Karasulu says:
+right I think we're on the same page
+
+Alex Karasulu says:
+I have a small idea though
+
+Alex Karasulu says:
+Basically wrt the codec's interfaces
+
+Alex Karasulu says:
+To me you give an array of bytes in a byte[] or a ByteBuffer (this is the 
+delivered partial chunk) and you get back a set of TLVs for that chunk.
+
+Alex Karasulu says:
+or take it in the opposite direction for a encoder
+
+Alex Karasulu says:
+this is your stage 1 (BER bytes ->TLVs)
+
+Alex Karasulu says:
+now we need to find a way to represent TLVs in a linear fashion and still 
+maintain the tree structure.  However we don't want direct back references 
+to where the list of TLVs plug into the entire tree because this would mean 
+we have to have the whole tree in memory.
+
+Alex Karasulu says:
+does that make sense I know its a lil nebulous
+
+Wes says:
+Keep it simple  
+
+Alex Karasulu says:
+ok in decoding bytes go in and TLVs come out
+
+Wes says:
+Right.
+
+Alex Karasulu says:
+state is maintained between times u pump in bytes
+
+Alex Karasulu says:
+wit me?
+
+Wes says:
+Yup.
+
+Alex Karasulu says:
+now the TLVs comming out are a peice of the TLV tree
+
+Wes says:
+You got to be able to handle partial Ts, Ls, and Vs.
+
+Alex Karasulu says:
+right that's part of the state stuff
+
+Alex Karasulu says:
+if you're stuck in the middle of a simple tlv then you don't pump it out until 
+the chunks to complete it have arrived
+
+Alex Karasulu says:
+wit me?
+
+Wes says:
+right.
+
+Alex Karasulu says:
+So the key here is to have the right TLV represntation or data structure.  We 
+have some requirements on this.
+
+Alex Karasulu says:
+the TLVs that come out of the decoder cannot directly, with java references, 
+refer to other TLVs  that came out before.  Because these references would 
+require the entire TLV tree in memory.
+
+Alex Karasulu says:
+This is one of those requirements you agree?
+
+Wes says:
+I don't see that being an issue.
+
+Wes says:
+The parent needs to know about the children, but not vis a versa.
+
+Alex Karasulu says:
+right
+
+Wes says:
+and I don't see how you are going to be able to assemble an ASN.1 message in a 
+state driven fashion without making it very complicated.
+
+Alex Karasulu says:
+that's our primary issue here
+
+Wes says:
+and have two decoders hooked together as well.
+
+Alex Karasulu says:
+its a big problem to overcome
+
+Alex Karasulu says:
+and do it elegantly
+
+Alex Karasulu says:
+If we do this then our BER ASN.1 codec will be hot working in a non-blocking 
+fashion and being very efficient.  It's like the way SAX is used for reading 
+XML for our ASN.1 messages instead of using DOM.
+
+Alex Karasulu says:
+the ideas are similar
+
+Alex Karasulu says:
+you didn't think this was gonna be a cake walk did ya  
+
+Wes says:
+Hmmmm.
+
+Alex Karasulu says:
+you do understand where I was coming from wit the sax and dom stuff right?
+
+Wes says:
+yea.
+
+Wes says:
+That I understand.
+
+Alex Karasulu says:
+do you think its possible?
+
+Wes says:
+So you have an event driven ASN.1 parser.
+
+Wes says:
+I think that's still easy.
+
+Wes says:
+However, assembling them into the messages is still complicated.
+
+Wes says:
+every ASN.1 message type would have to be derived from our parser.
+
+Wes says:
+Then a factory could create the message type based on the application type.
+
+Alex Karasulu says:
+hmmm
+
+Alex Karasulu says:
+what do you mean by: "every ASN.1 message type would have to be derived from 
+our parser.
+
+Wes says:
+You want the ASN.1 messages to be able to assemble themselves? or no.
+
+Alex Karasulu says:
+Now you're talking about using the ASN.1 specification like a DTD to drive the 
+decoding
+
+Alex Karasulu says:
+?
+
+Alex Karasulu says:
+Yep I see yes
+
+Alex Karasulu says:
+u use the ASN.1 spec or a set of classes generated by an ASN.1 spec compiler
+
+Alex Karasulu says:
+question is do we need a compiler now?
+
+Wes says:
+Right.
+
+Wes says:
+Factory returns the ASN.1 message on the application tag.
+
+Alex Karasulu says:
+right I see where your going with the design
+
+Wes says:
+the parser then passes everything to the ASN.reader interface,
+
+Wes says:
+SAX like.
+
+Alex Karasulu says:
+Hmm sounds like it should be very possible
+
+Wes says:
+of the application object.
+
+Wes says:
+who knows how to assemble himself.
+
+Alex Karasulu says:
+right
+
+Alex Karasulu says:
+This is huge
+
+Alex Karasulu says:
+I wonder if other ASN.1 tools have this sax like mechanism already in place.
+
+Wes says:
+But how do we handle ASN.1 messages which need to be streamed.
+
+Wes says:
+like a huge search result.
+
+Alex Karasulu says:
+that's not so much the issue 
+
+Alex Karasulu says:
+a large result set takes n+2 messages
+
+Alex Karasulu says:
+sorry n+1
+
+Wes says:
+You have a search result tight.
+
+Wes says:
+Tag = Applicationz
+Length = 00
+Value = Search Results
+
+Wes says:
+Now V is made up of thousands of result messages.
+
+Alex Karasulu says:
+In the LDAP protocol a search result is returned as n+1 messages.
+
+Alex Karasulu says:
+each result is an SearchEntryResponse for the 'n' and one SearchDoneResponse 
+PDU to end the resultset
+
+Alex Karasulu says:
+n+1 messages
+
+Wes says:
+Ah.
+
+Wes says:
+But are they wrapped in an application TLV?
+
+Alex Karasulu says:
+but think of a large blob of data
+
+Wes says:
+or is it just one stream of TLVs.
+
+Alex Karasulu says:
+like say some binary chunk
+
+Alex Karasulu says:
+the application TLV for each response type is in the LDAP message envelope.  
+There is a top level LDAP message type which is a TLV then the different 
+response types have you know some enumeration values to determine which 
+response type the top level envelope or application TLV represents
+
+Wes says:
+Right.
+
+Alex Karasulu says:
+but your question is valid for say a single SearchEntryResponse where one of 
+the attributes is a huge binary chunk
+
+Wes says:
+So the event firing for the top level envelope will be different than the TLVs 
+which are part of the envelope.
+
+Alex Karasulu says:
+the top level LDAPMessage envelope defined for the LDAP asn.1 will be a 
+constructred TLV
+
+Alex Karasulu says:
+event might fire for it
+
+Alex Karasulu says:
+same one every time
+
+Wes says:
+Right, but not after the entire TLV is read into memry.
+
+Wes says:
+that would defeat our SAX based parser.
+
+Alex Karasulu says:
+but its constitution will change depending on the type of message it is
+
+Alex Karasulu says:
+right
+
+Alex Karasulu says:
+exactly
+
+Wes says:
+I'm with you.
+
+Alex Karasulu says:
+you would get a start_ldap_message event
+
+Wes says:
+Actually,
+
+Wes says:
+for the envelope, you would need to hit the factory.
+
+Wes says:
+to get the appropriate LDAP message.
+
+Alex Karasulu says:
+then perhaps the message_type_event will fire to note the contained TLV that 
+specifies the LDAP application's message type.
+
+Alex Karasulu says:
+et. cetera. see where i'm going with it - you don't need the entire message to 
+fire its arrival.  Like sax where you say start tag for this element then the 
+contained elemenets then close tags etc.
+
+Wes says:
+Got ya.
+
+Wes says:
+I think that's pretty cool.
+
+Alex Karasulu says:
+I think we're getting somewhere cool here I'm very excited.  I need to take 
+another look at a sax implementation again out there.  It will give me some 
+insight into some possible general architecture for us.
+
+Alex Karasulu says:
+Now going back to the massive chunk of binary.  So we have a 
+SearchEntryResponse with an entry of the result set containing an attribute 
+that is a huge binary chunk.  How do we stream it out right?  Then we can 
+talk about how we stream it in.
+
+Alex Karasulu says:
+Streaming it out is easy.  Let's for a moment presume that we can actually 
+stream out of the jdbm stuff.  You basically convert the long known length BER 
+encoding to the indeterminant encoding.  Then send out individual chunks of 
+this binary attribute in separate TLVs.  So you're turning big assed primitive 
+TLVs into constructed TLVs chunking out the content hence not needing the 
+entire V in memor
+
+Alex Karasulu says:
+y.
+
+Wes says:
+That's fine for us.  We have control over the encoding.
+
+Wes says:
+We won't be so lucky on the inbound side.
+
+Alex Karasulu says:
+Right
+
+Alex Karasulu says:
+Now let's think about that beast.
+
+Alex Karasulu says:
+We have a binary -> tlv encoder spitting out tlvs with each bit of input
+
+Alex Karasulu says:
+meant decoder above sorry
+
+Alex Karasulu says:
+now if the indeterminate length is used by the client when encoding and 
+sending to the server the server is ok the data is already chopped up and 
+its all good.  If not and the long length encoding is used then the data 
+comes into the server's decoder in chunks but the decoder sees a hugh 
+long length. 
+
+Alex Karasulu says:
+Based on some threshold the decoder translates the incoming long length and 
+values for the simple type (primitive TLV) into a constructed TLV breaking 
+up the large know length TLV into the indeterminant form which can be spit 
+out with a few nested TLVs at a time (with each input chunk going into the 
+decoder).
+
+Alex Karasulu says:
+You follow? Decoder automatically breaks up large primitive long length encoded 
+TLVs into the indeterminate form and spits those out in peices rather than the 
+one large primitive TLV.
+
+Wes says:
+What does that buy us?
+
+Alex Karasulu says:
+streaming
+
+Wes says:
+Is not the ASN message gonna re-assemble it anyways.
+
+Wes says:
+Do you still end up with 200K picture in the ASN.1 message.
+
+Alex Karasulu says:
+yeah that's application specific - remember we're talking just the BER->TLV 
+codec
+
+Alex Karasulu says:
+the other codec is Type to TLV
+
+Wes says:
+If we are using a SAX based parser, then the Type will be assembling itself as 
+the TLVs are decoded and fired.
+
+Alex Karasulu says:
+keeping it streaming means you don't have 2X the data or 400K in use just to 
+get the 200K picture
+
+Alex Karasulu says:
+right
+
+Wes says:
+At some point, you are going to have to put your faith in the garbage collector.
+
+Alex Karasulu says:
+right but that's not in the codec BER to TLV code
+
+Alex Karasulu says:
+keep that lean and mean - why you ask
+
+Wes says:
+Also, if you want a truly small memory footprint, then you could put stuff like 
+that in a small embedded database.
+
+Alex Karasulu says:
+well the TLV to Type code can be made lean and mean too
+
+Wes says:
+I just don't think at this stage that we need to be all that worried about huge 
+blocks of binary data.
+
+Alex Karasulu says:
+right we use referrals to data on disk to manage large peices of data that 
+needsto be streamed but this we can do later.
+
+Wes says:
+Exactly.
+
+Alex Karasulu says:
+yes but we want the options to be open - right now we can just design the 
+interfaces so all this can be added later.
+
+Alex Karasulu says:
+Interfaces and contracts should be designed to allow these very low memory 
+footprints.  Thinking through the process and what it takes to get there 
+makes us understand better what the design and interfaces should look like.
+
+Alex Karasulu says:
+I don't care if the first implementation is a hog
+
+Wes says:
+The BER stuff today doesn't deal with this.
+
+Wes says:
+It doesn't care.
+
+Alex Karasulu says:
+for large peices of data
+
+Wes says:
+It's an application issue.
+
+Alex Karasulu says:
+right
+
+Alex Karasulu says:
+what the app does with it is upto the app but lets keep the ber codecs low in 
+memory image regardless of the fact that some app will be a pig and stream the 
+data into memory anyway.  This is all that I'm trying to say.
+
+Alex Karasulu says:
+wit me?
+
+Wes says:
+K.
+
+Alex Karasulu says:
+cool we're tight on this but I think it will take more research on both our 
+parts - anyway apache is back up again after a power failure.  Here's the new 
+stuff I created for ya:   
+http://cvs.apache.org/viewcvs.cgi/incubator/directory/snickers/?root=Apache-SVN
+
+Alex Karasulu says:
+that's the top level of the snickers (snacc replacement) subproject
+
+Alex Karasulu says:
+that's all you and Jeff with the C based version of this thang
+
+Wes says:
+Right.
+
+Wes says:
+You won't find much other ASN.1 stuff out there.
+
+Wes says:
+I'm comfortable that no one is doing it this way, either.
+
+Wes says:
+It will make it unqiuely, Apache.
+
+Alex Karasulu says:
+Ok. Let's touch base in a day or two to regroup
+
+Wes says:
+Do you think ASN.1 is going to die?
+
+Alex Karasulu says:
+this is all good stuff and I'll try to get it out there.
+
+Alex Karasulu says:
+no way
+
+Alex Karasulu says:
+ASN.1 is awesome stuff
+
+Wes says:
+We'll see.
+
+Alex Karasulu says:
+SNMP is based on it and so is Kerberose
+
+Alex Karasulu says:
+what's the alternative?
+
+Wes says:
+XML is what everyone is using now.
+
+Alex Karasulu says:
+well there is XER for ASN.1 
+
+Alex Karasulu says:
+XML Encoding Rules
+
+Alex Karasulu says:
+ASN.1 can go to BER, PER, XER, and DER
+
+Wes says:
+Yes.
+
+Alex Karasulu says:
+the encoding does not effect the ASN.1 specification and that is what makes 
+ASN.1 a winner always.
+
+Wes says:
+Slapping XML on ASN.1 ain't the same.
+
+Alex Karasulu says:
+the XML format is just for the encoding of the data types 
+
+Wes says:
+I agree that ASN.1 is a good protocol.
+
+Alex Karasulu says:
+protocol specification syntax
+
+Alex Karasulu says:
+it kicks ass I think and is here to stay.
+
+Wes says:
+If we do this, we are going to go backwards right?
+
+Wes says:
+Do the compiler last.
+
+Alex Karasulu says:
+go backwards?
+
+Alex Karasulu says:
+yeah that might be the case or we can work it together.
+
+Wes says:
+You need to let me work this.
+
+Alex Karasulu says:
+I can do the compiler with you and you can handle the runtime
+
+Wes says:
+You got other things to do.
+
+Alex Karasulu says:
+ok its all you then 
+
+Alex Karasulu says:
+I'm just a follower
+
+Wes says:
+I won't mind help with the compiler.
+
+Wes says:
+Just don't get going on it any time soon  
+
+Alex Karasulu says:
+sure I have extensive javacc and antlr experience
+
+Wes says:
+Deal.
+
+Alex Karasulu says:
+hehe no worries with that my plate as you know is overflowing.
+
+Alex Karasulu says:
+my bladder too
+
+Alex Karasulu says:
+I'll catch ya later I need to hit the head
+
+Wes says:
+Talk about the decoder's stream.
+
+Wes says:
+K
+
+Alex Karasulu says:
+ttyl
+
+Wes says:
+Talk later then.
+
+Alex Karasulu says:
+ok gimme 45 seconds
+
+Alex Karasulu says:
+I'm back
+
+Alex Karasulu says:
+what about the decoder's stream.
+
+Wes says:
+So, how do we feed the decoder then.
+
+Alex Karasulu says:
+Its all about how we design our interfaces.  You know I've been looking at 
+commons-codec and see some potential but changes will be needed.
+
+Alex Karasulu says:
+Follow me for a sec.
+
+Alex Karasulu says:
+Now the codec interfaces are designed to convert stuff in one shot.  
+bytes in bytes out sort of thang.  Very blocking dependent stuff and not very 
+cool for us with a SEDA and NIO based server.  
+
+Alex Karasulu says:
+wit me?
+
+Wes says:
+right.
+
+Alex Karasulu says:
+As you might have guessed this is not good for servers that need to keep 
+memory footprints low while servicing possible serveral hundred requests 
+per second.
+
+Alex Karasulu says:
+So what do we do? We design new non-blocking and NIO based interfaces for the 
+codec API and submit them.  
+
+Alex Karasulu says:
+its down again damn
+
+Wes says:
+I got my update  
+
+Alex Karasulu says:
+cool
+
+Wes says:
+Must of brought it down.
+
+Alex Karasulu says:
+yeah maybe it will be up soon
+
+Alex Karasulu says:
+anyway
+
+Alex Karasulu says:
+We redesign these codec interfaces to manage an encoding session and a decoding
+session so chunks can be process in a stateful manner to be conducive to 
+non-blocking use.
+
+Alex Karasulu says:
+Or we use events like you said
+
+Alex Karasulu says:
+Basically we contribute this to the commons stuff and make sure the community 
+understands why and what we're doing.  That way they can double check us.
+
+Alex Karasulu says:
+Then we use those interfaces to implement the ASN.1 stuff.
+
+Wes says:
+Right.
+
+Alex Karasulu says:
+We do this in the snickers area but put back as much into the commons codec as 
+we can.  You game with this strategy?
+
+Wes says:
+Yea, that's fine.
+
+Wes says:
+I'll check out commons code as soon as it comes up.      
+
+</source>
+     </subsection>
+   </section>
+  </body>
+</document>

Added: incubator/directory/asn1/branches/rewrite/ber/xdocs/index.xml
URL: http://svn.apache.org/viewcvs/incubator/directory/asn1/branches/rewrite/ber/xdocs/index.xml?view=auto&rev=154725
==============================================================================
--- incubator/directory/asn1/branches/rewrite/ber/xdocs/index.xml (added)
+++ incubator/directory/asn1/branches/rewrite/ber/xdocs/index.xml Mon Feb 21 13:44:46 2005
@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<document>
+  <properties>
+    <author email="akarasulu@apache.org">Alex Karasulu</author>
+    <title>BER Runtime</title>
+  </properties>
+  <body>
+    <section name="Introduction">
+      <subsection name="What is it?">
+      <p>
+        The BER Runtime is an API for encoding and decoding ASN.1
+        data structures using Basic Encoding Rules (BER).  It implements
+        extentions to the <a href="http://jakarta.apache.org/commons/codec">
+        commons-codec</a> API, for building stateful chunking encoder decoder
+        pairs that maintain state between processing calls.
+      </p>
+      </subsection>
+
+      <subsection name="Stateful Codecs">
+      <p>
+        More information on these new codec interfaces are availabled on the
+        <a href="../asn1-codec/index.html">stateful codec</a> home page.
+        You might want to read this before you continue since these extentions
+        are the basis to all ASN.1 encoders and decoders.
+      </p>
+      </subsection>
+
+      <subsection name="What is encoded/decoded?">
+      <p>
+        The BER runtime is protocol or ASN.1 module independent.  The unit of
+        substrate is a BER TLV (Tag, Length, Value) so any BER based protocol
+        can be decoded and encoded by the BER codec to and from TLV tuples.
+      </p>
+      </subsection>
+    </section>
+
+    <section name="BER Codec User Guides and Design Documents">
+      <table>
+        <tr>
+          <th>Subject</th>
+          <th>Description</th>
+        </tr>
+
+        <tr>
+          <td><a href="./asn1berinfo.html">ASN.1 and BER Information</a></td>
+          <td>Links to various books and specification on ASN.1 and BER</td>
+        </tr>
+
+        <tr>
+          <td><a href="./BERDecoderDesign.html">BER Decoder Design</a></td>
+          <td>Explains how and why the BERDecoder was designed</td>
+        </tr>
+
+        <tr>
+          <td><a href="./BERDigesterDesign.html">BER Digester Design</a></td>
+          <td>Explains how and why the BERDigester was designed</td>
+        </tr>
+
+        <tr>
+          <td><a href="./BEREncoderDesign.html">BER Encoder Design</a></td>
+          <td>Explains how and why the BEREncoder was designed</td>
+        </tr>
+
+      </table>
+    </section>
+  </body>
+</document>

Added: incubator/directory/asn1/branches/rewrite/codec/xdocs/index.xml
URL: http://svn.apache.org/viewcvs/incubator/directory/asn1/branches/rewrite/codec/xdocs/index.xml?view=auto&rev=154725
==============================================================================
--- incubator/directory/asn1/branches/rewrite/codec/xdocs/index.xml (added)
+++ incubator/directory/asn1/branches/rewrite/codec/xdocs/index.xml Mon Feb 21 13:44:46 2005
@@ -0,0 +1,326 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<document>
+  <properties>
+    <author email="akarasulu@apache.org">Alex Karasulu</author>
+    <title>Stateful Codecs</title>
+  </properties>
+  <body>
+    <section>
+    <subsection name="Introduction">
+      <p>
+        Codecs are bidirectional data transformations.  The data transformed,
+        often referred to as the substrate, may be [en]coded or decoded hence
+        the word codec.  The word codec also refers to the actual software
+        used to encode and decode data.  We use the term stateful codec for
+        lack of a better description for encoder/decoder pairs possessing
+        certain abilities and exhibiting the following behavoirs:
+      </p>
+
+      <ul>
+        <li>the ability to interrupt and resume operation without loosing
+            state</li>
+        <li>the ability to process a substrate in one or more steps operating
+            on small chunks rather than all of it in one large operation</li>
+        <li>free up resources while not actively processing perhaps until more
+            of the substrate is available, or just to multiplex limited
+            resources</li>
+        <li>use a small fixed size chunk buffer rather than a variable sized
+            buffer equal to the entire size of the substrate what ever that
+            may be</li>
+      </ul>
+    </subsection>
+
+    <subsection name="Advantages">
+      <p>
+        The abilities or behavoirs listed above make stateful codecs ideal for
+        use in resource critical situations.  Servers for example based on
+        codecs may have to perform several thousand concurrent encode/decode
+        operations.  The resources required for such operations, namely threads
+        and memory buffers will be limited.  Most of the time these operations
+        will be waiting for IO to complete so they can free up resources to
+        allow other operations to proceed.  Stateful codecs make this possible
+        and complement servers designed using non-blocking IO constructs.
+      </p>
+
+      <p>
+        Servers cannot afford to allocate variable sized buffers for arriving
+        data.  Allowing variable sized buffers based on incoming data
+        sizes opens the door for DoS attacks where malicious clients can
+        cripple or crash servers, by pumping in massive or never ending
+        data streams.  Stateful codecs enable fixed size processing overheads
+        regardless of the size of the data unit transmitted to the server.
+        Smaller codec footprints lead to smaller server process memory
+        footprints.
+      </p>
+
+      <p>
+        These advantages also make stateful codecs ideal for use in resource
+        limited environments like embedded systems, PDAs or cellular phones
+        which use ASN.1 and one of its encoding schemes to control data
+        transmission.  These systems all run on limited resources where the
+        codec's operational footprint will have dramatic effects on the
+        performance of the device.
+      </p>
+    </subsection>
+
+    <subsection name="How is a stateful codec defined?">
+      <p>
+        There are several ways to skin this cat.  To this day discussions are
+        underway at the ASF to determine the best approach.  Until a consensus
+        is reached we have decided to use an event driven approach where the
+        events are modelled as callbacks.  To better explain the approach we
+        need to discuss it within the context of encoding/decoding.
+      </p>
+
+      <p>
+        Depending on the operation being performed, available chunks of the
+        substrate are are processed using either the <code>encode()</code> or
+        the <code>decode()</code> method.  These methods hence are presumed
+        to process small chunks of the substrate.  The specific codec
+        implementation should know how to maintain state based on the encoding
+        between these calls to process a unit of substrate which likewise is
+        determined by the encoding.  So the encoding (a.k.a. codec) defines
+        what a unit of substrate is as well as any state information required
+        while peice-meal processing the substrate.  Several calls to these two
+        methods may be required to process a unit of the substrate.  When the
+        entire unit has been processed an event is fired.  Again the specific
+        codec detects the compete processing of a unit of substrate so it
+        knows when to fire this event.
+      </p>
+
+      <p>
+        Going back to our approach for defining a stateful codec, we modeled
+        the event as a callback to a specific interface.  For decoders this
+        would be a <code>DecoderCallback.decodeOccurred()</code> and for
+        encoders it would be an <code>EncoderCallback.encodeOccurred()</code>
+        method call.  These interface methods are called when an entire unit
+        of substrate is respectively decoded or encoded.
+      </p>
+
+      <p>
+        This approach also allows for codec chaining in a pipeline where
+        codecs may be stacked on top of one another.  The callback interfaces
+        are used to bridge together codecs by feeding the output of one codec
+        operation into the input of another.  Specific classes have been
+        included in the API to accomodate this usage pattern.
+      </p>
+
+      <center>
+        <img src="../images/all-uml.gif"/>
+      </center>
+
+    </subsection>
+
+    <subsection name="StatefulDecoder Usage">
+      <p>
+        StatefulDecoders use callbacks to notify the successful decode of a
+        unit of encoded substrate.  Other than this, the definition of what a
+        'unit of encoded substrate' is, depends on the codec's decoder
+        implementation.  The definition may be size constrained or be a
+        function of context.
+      </p>
+      
+      <p>
+        Basically you give a decoder some of the substrate every so often
+        as more of the substrate is made available, then when a unit of 
+        encoded substrate has been decoded, the decoder notifies those 
+        concerned by invoking the callback.  
+      </p>
+      
+      <p>
+        A demonstration of how a StatefulDecoder works is illustrated below:
+      </p>
+      
+      <source>
+StatefulDecoder decoder = new SomeConcreteDecoder( 512 ) ;
+DecoderCallback cb = new DecoderCallback() {
+  decodeOccurred( StatefulDecoder decoder, Object decoded ) {
+      // do something with the decoded object
+  }
+};
+decoder.setCallback( cb ) ;
+      </source>
+      
+      <p>
+        The StatefulDecoder uses a callback to deliver decoded objects which 
+        are the decoded 'unit of encoded substrate'.  StatefulDecoders are ideal
+        for use in high performance servers based on non-blocking IO.   Often
+        StatefulDecoders will be used with a Selector in a loop to detect input
+        as it is made available.  As the substrate arrives, it is be fed to
+        the decoder intermittantly.  Finally the callback delivers the decoded
+        units of encoded substrate.  Below there is a trivialized example of
+        how a StatefulDecoder can be used to decoded the substrate as it
+        arrives fragmented by the tcp/ip stack:
+      </p>
+      
+      <source>
+while ( true ) {
+  ...
+  SelectionKey key = ( SelectionKey ) list.next() ;
+  if ( key.isReadable() ) {
+    SocketChannel channel = ( SocketChannel ) l_key.channel() ;
+    channel.read( buf ) ;
+    buf.flip() ;
+    decoder.decode( buf ) ;
+  }
+  ...
+}
+      </source>
+      
+      <p>
+        As you can see from the code fragment the decode() returns nothing
+        since it has a void return type.  Because the callback is used to
+        deliver the finished product when it is ready, the decode operation
+        can occur asynchronously in another thread or stage of a server if
+        desired.
+      </p>
+    </subsection>
+    
+    <subsection name="Strengths and Weaknesses">
+      <p>
+        As can be seen from the section above and some of the characteristics 
+        of StatefulDecoders, they are ideal for building network servers.  These
+        decoders waste very little memory per request, cannot be overloaded by
+        massive requests which may be used for DoS attacks, and they process the
+        substrate as it arrives in chucks instead of in one prolonged CPU and 
+        memory intensive step.
+      </p>
+      
+      <p>
+        Servers with a high degree of concurrency need to keep overheads low.
+        StatefulDecoders certainly help achieve that end by keeping the
+        active processing footprint low with a constant size regardless of the 
+        size of the substrate.
+      </p>
+      
+      <p>
+        The cost of creating a decoder for every new connection is usually
+        very minimal however we cannot forsee every possible implementation.
+        Regardless of the cost associated with dedicating a StatefulDecoder
+        to each new connection, stateful protocol servers will often benefit
+        most, as opposed to a stateless server.  The reasoning is as follows:
+        the longer the life of the connection, the more worth while it
+        is to create a StatefulDecoder and thereby have it amortize over the 
+        life of the connection.
+      </p>
+      
+      <p>
+        The primary drawback is that StatefulDecoders are much more complex to
+        implement.  They are basically state driven automata which change
+        their state with the arrival of data.  Furthermoe it is very difficult
+        for StatefulDecoders to gracefully recover from corrupt or lost input.
+      </p>
+    </subsection>
+    
+    <subsection name="StatefulDecoder Chaining/Stacking">
+      <p>
+        StatefulDecoders can easily be chained or stacked to operate on a 
+        substrate stream.  This is achieved by having the callback of one 
+        decoder feed the <code>decode(Object)</code> method of another.  Hence
+        the decoded byproduct of one decoder is the encoded substrate of 
+        another.
+      </p>
+      
+      <p>
+        Because the occurence of chaining may be common and several folks have
+        already expressed their interest in it, we have devised a special
+        StatefulDecoder implementation called a DecoderStack.  It itself is 
+        a decoder however other decoders can be pushed onto it.  When empty
+        without any decoders in the stack it operates in pass-thro mode.  The
+        decode operation is basically the identity transformation.  When
+        StatefulDecoders are pushed, decode operations invoke a chain of
+        decoders starting with the bottom most in the stack going up to the
+        top.  The final callback invoked is the callback registered with the
+        DecoderStack.
+      </p>
+      
+      <p>
+        Below is an example of how this DecoderStack is used.  The example is
+        taken from one of the JUnit test cases for DecoderStack:
+      </p>
+
+      <source>
+public void testDecode() {
+  DecoderStack stack = new DecoderStack() ;
+  CallbackHistory history = new CallbackHistory() ;
+  stack.setCallback( history ) ;
+  stack.push( decoder ) ;
+  stack.decode( new Integer(0) ) ;
+  assertEquals( new Integer(0), history.getMostRecent() ) ;
+        
+  stack.push( new IncrementingDecoder() ) ;
+  stack.decode( new Integer(0) ) ;
+  assertEquals( new Integer(1), history.getMostRecent() ) ;
+
+  stack.push( new IncrementingDecoder() ) ;
+  stack.decode( new Integer(0) ) ;
+  assertEquals( new Integer(2), history.getMostRecent() ) ;
+}
+...
+
+class IncrementingDecoder extends AbstractStatefulDecoder
+{
+  public void decode( Object encoded ) throws DecoderException
+  {
+    Integer value = ( Integer ) encoded ;
+    value = new Integer( value.intValue() + 1 ) ;
+    super.decodeOccurred( value ) ;
+  }
+}
+      </source>      
+    </subsection>
+    
+    <subsection name="Recommendations to Implementors">
+      <p>
+        Keep it simple and rely on chaining to divide and concur complex 
+        decoders into several trivial decoders.  Besides simple chaining,  
+        situations will warrent the use of a choice driven decoder.  Such a 
+        decoder chooses which subordinate decoder to use based on its
+        current state.  For example in the simple BER byte stream to TLV 
+        decoder in Snickers, their is a TagDecoder, a LengthDecoder and
+        several Value decoders that are swapped in and out when the top 
+        BERDecoder switches state or detects a new primitive datatype.
+      </p>
+      
+      <p>
+        When reading encoded data from buffers, keep in mind that there are 
+        5 different possible configurations to the contents of arriving data 
+        with respect to the unit of encoded substrate:
+      </p>
+
+      <!--
+        todo add illustrations using images here - its not that hard
+        might want to turn this into a table instead of a ul if we decide
+        to do that
+      -->
+
+      <ul>
+        <li>
+          it contains a single complete discrete unit of encoded substrate
+        </li>
+        <li>
+          it contains many discrete and complete units of encoded substrate
+        </li>
+        <li>
+          it contains a partial fragment of a unit of encoded substrate
+        </li>
+        <li>
+          it contains two partial fragments of a unit of encoded substrate with
+          the start of one and the end of another
+        </li>
+        <li>
+          it contains one or more fragments with one or more units of encoded 
+          substrate
+        </li>
+      </ul>
+      
+      <p>
+        When fragments arrive they are either head or tail fragments.  Head
+        fragments are those that start a unit and they are found at the end 
+        of the buffer.  Tail fragments end a unit of encoded substrate and are
+        found at the front of the buffer.
+      </p>
+    </subsection>
+    </section>
+  </body>
+</document>

Added: incubator/directory/asn1/branches/rewrite/stub-compiler/xdocs/index.xml
URL: http://svn.apache.org/viewcvs/incubator/directory/asn1/branches/rewrite/stub-compiler/xdocs/index.xml?view=auto&rev=154725
==============================================================================
--- incubator/directory/asn1/branches/rewrite/stub-compiler/xdocs/index.xml (added)
+++ incubator/directory/asn1/branches/rewrite/stub-compiler/xdocs/index.xml Mon Feb 21 13:44:46
2005
@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<document>
+  <properties>
+    <author email="akarasulu@apache.org">Alex Karasulu</author>
+    <title>Snickers ASN.1 Java Stub Compiler</title>
+  </properties>
+  <body>
+    <section name="Coming soon ...">
+      <p>
+        Wonderful things are coming soon ...
+      </p>
+    </section>
+  </body>
+</document>



Mime
View raw message