qpid-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Qpid > Protocol Engines
Date Mon, 11 Feb 2013 12:12:00 GMT
<html>
<head>
    <base href="https://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/2042/9/21/_/styles/combined.css?spaceKey=qpid&amp;forWysiwyg=true"
type="text/css">
    </head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
    <h2><a href="https://cwiki.apache.org/confluence/display/qpid/Protocol+Engines">Protocol
Engines</a></h2>
    <h4>Page <b>edited</b> by             <a href="https://cwiki.apache.org/confluence/display/~rhs">Rafael
H. Schloming</a>
    </h4>
        <br/>
                         <h4>Changes (12)</h4>
                                 
    
<div id="page-diffs">
                    <table class="diff" cellpadding="0" cellspacing="0">
    
            <tr><td class="diff-changed-lines" >The Proton toolkit is built around
the concept of a protocol engine. The idea behind a protocol engine is to capture all the
complex details of implementing a given protocol in a way that is as decoupled as possible
from OS details such as I/O and threading <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">models,
resulting in</span> <span class="diff-added-words"style="background-color: #dfd;">models.
The result is</span> a highly <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">reusable/embeddable</span>
<span class="diff-added-words"style="background-color: #dfd;">portable and easily embedded</span>
component that provides a full protocol implementation. To see how this works in the general
case, it helps to start out with some simple examples of engines and build up to the full
model used by Proton. <br></td></tr>
            <tr><td class="diff-unchanged" > <br></td></tr>
            <tr><td class="diff-changed-lines" >A common first step in writing
networking code is to debug the network I/O portion independently from the rest of the application
by using a much simpler protocol than the application will ultimately demand. Often the simple
protocol chosen is the echo protocol. Anything sent to an echo server will be copied directly
back to the client. The exact details of implementing an echo server might vary depending
on what kind of I/O framework you are using, but it&#39;s inevitable that somewhere in
the heart of an echo server there is some kind of data structure, either implicit or explicit,
serving as a byte queue <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">aka
a</span> <span class="diff-added-words"style="background-color: #dfd;">or</span>
circular buffer. If you define an abstract interface around this <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">byte
queue,</span> <span class="diff-added-words"style="background-color: #dfd;">data
structure,</span> you have something that captures the admittedly trivial semantics
of the echo protocol, but is a pure data structure, and therefore doesn&#39;t need to
interact with the OS at all. We now have the echo engine depicted below. A simple circular
buffer or byte queue, but with a slightly fancier <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">title
and because the interface is abstract,</span> <span class="diff-added-words"style="background-color:
#dfd;">title. With this abstract interface, however,</span> we can start tweaking
the implementation <span class="diff-added-words"style="background-color: #dfd;">of
our engine</span> independently from the rest of our server. <br></td></tr>
            <tr><td class="diff-unchanged" > <br>{gliffy:name=Echo Engine|align=center|size=L|version=2}
<br> <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">Now
let&#39;s try to make this example do something a little more interesting by adding a
query interface to our engine as depicted below. The I/O portion of our application still
views the engine as a pure circular buffer, however we can now add more sophisticated logic
based on inspecting the stream of bytes that flows through the engine. We could, for example,
log exactly how many bytes pass through, or decode some or all of the stream and provide an
interface that would allow the application to log the semantic data encoded within. <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">Let&#39;s
start by making our example do something a little more interesting. We can add a query interface
to our engine as depicted below. The I/O portion of our application still views the engine
as an ordinary circular buffer, however we can now add application logic that uses the query
interface to take action. The exact nature of the query interface depends on the protocol
and on the engine internals. A simplistic query interface might just provide a byte count
of data passing through, whereas a more sophisticated engine could decode and make available
semantic content embedded within the byte stream. While a little bit more capable than a simple
echo engine, this kind of engine is still just a simple data structure with some added interrogatives,
and so retains the desired decoupling and portability. <br></td></tr>
            <tr><td class="diff-unchanged" > <br>{gliffy:name=Echo Engine
with Query Interface|align=center|size=L|version=2} <br> <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">This
is about as far as we can go with an echo protocol, so lets now consider modifying our example
to be a simple request/response protocol. Something where every response is a predefined function
of each request. As depicted below, with the transformation function entirely encapsulated
inside the engine, the interface presented to the I/O portion of the application is completely
identical to our original echo server. However, unlike the echo server, we now have the ability
to capture a large class of protocols inside this kind of engine, and furthermore as the engine
is still essentially a pure data structure (if not quite a simple one anymore) it remains
entirely decoupled from aspects of the OS such as I/O and threading. <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">Now
consider modifying our example to be a simple request/response protocol where every response
is a predefined function of each request. We can provide an engine for this kind of protocol
by modifying our simple echo engine to detect requests and replace them with responses. With
the transformation function entirely encapsulated inside the engine, the interface presented
is completely identical to our original echo engine. However, unlike the echo engine, we now
have the ability to capture a larger class of protocols. <br></td></tr>
            <tr><td class="diff-unchanged" > <br>{gliffy:name=Simple Request-Response
Engine|align=center|size=L|version=2} <br> <br></td></tr>
            <tr><td class="diff-changed-lines" >Now just as we did with the echo
server, we can also extend the simple request/response scenario with a query interface as
depicted below. <span class="diff-added-words"style="background-color: #dfd;">And as
before, our engine is still just a data structure with well defined interfaces that keep it
decoupled and portable.</span> <br></td></tr>
            <tr><td class="diff-unchanged" > <br>{gliffy:name=Simple Request-Response
Engine with Query Interfac|align=center|size=L|version=2} <br> <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">At
this point even though we could capture a wide variety of protocols with this class of engine,
there are still significant limitations. For example if the response transformation requires
accessing a file as is the case with classic HTTP, our engine is no longer decoupled from
the OS, and could in fact block on file I/O, or suppose in the case of modern HTTP the request
might actually be invoking application logic and we simply can&#39;t pre-define the result
within the confines of our library. To accommodate these scenarios, while still keeping the
engine as a simple data structure that is entirely decoupled from the OS, we can add a compliment
to the query interface that allows the application to control the response as depicted below.
Note that in this case, the I/O portion of our server still views the engine as a simple circular
buffer. It need not know what kind of processing happens to the byte stream as it passes through
the engine. <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">Despite
being able to capture a large class of protocols, the simple request/response engine still
has some significant limitations. For example if the response transformation requires accessing
a file, as is the case with a classic HTTP implementation, our engine is no longer decoupled
from the OS, and could in fact block on file I/O. Also, if the request is actually invoking
application logic such as with an RPC protocol or with most modern usage of HTTP, we simply
can&#39;t pre-define the result within the confines of our library. To accommodate these
scenarios, we need to modify our engine a little further to allow the application to control
aspects of the response. To do this we add a control interface that can be used in conjunction
with the query interface to inspect a request and modify the response. <br></td></tr>
            <tr><td class="diff-unchanged" > <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">As
we do this we begin to see two distinct facets of the engine emerge. We have the interface
presented to the I/O portion of our server which remains identical to our original echo engine,
i.e. a simple circular buffer. Independent of this we have a higher level API emerging that
presents the semantics of the protocol to the part of the server dealing with application
logic. As we&#39;ve seen, the latter interface can evolve and change depending on the
nature of the protocol without any impact on the former. The I/O portion of our server need
not know what kind of processing happens to the byte stream as it passes through the engine.
<br> <br></td></tr>
            <tr><td class="diff-unchanged" >{gliffy:name=Request-Response Engine|align=center|size=L|version=3}
<br> <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >{gliffy:name=General Engine|align=center|size=L|version=3}
<br> <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">A
further refinement of the general purpose engine is to separate the control interface into
a distinct entity with its own lifecycle that can be bound to a given transport and then unbound.
This separation has a number of nice features. For one thing it partitions the engine interface
into a bottom half that can keep a very simple I/O oriented view of the world, and a top half
that reflects the high level application interface to the protocol. This separation also permits
the control interface to carry state across multiple physical connections, which can allow
for a much more sophisticated control entity that can provide a simpler interface. <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">A
further refinement of the general purpose engine is to separate the control interface into
a distinct entity with its own lifecycle. This entity can be bound to a given transport and
then unbound. This separation formalizes the two facets of the engine interface discussed
above into a bottom half that can keep a very simple I/O oriented view of the world, and a
top half that reflects the high level application interface to the protocol. The distinct
lifecycle also permits the control interface to carry state across multiple physical connections,
which can allow for a much more sophisticated control entity that can provide a simpler interface,
for example automatically recovering state from a previous connection as is done with HTTP
sessions. <br></td></tr>
            <tr><td class="diff-unchanged" > <br>{gliffy:name=Seperate Control
Interface Lifcycle|align=center|size=L|version=2} <br></td></tr>
    
            </table>
    </div>                            <h4>Full Content</h4>
                    <div class="notificationGreySide">
        <p>The Proton toolkit is built around the concept of a protocol engine. The
idea behind a protocol engine is to capture all the complex details of implementing a given
protocol in a way that is as decoupled as possible from OS details such as I/O and threading
models. The result is a highly portable and easily embedded component that provides a full
protocol implementation. To see how this works in the general case, it helps to start out
with some simple examples of engines and build up to the full model used by Proton.</p>

<p>A common first step in writing networking code is to debug the network I/O portion
independently from the rest of the application by using a much simpler protocol than the application
will ultimately demand. Often the simple protocol chosen is the echo protocol. Anything sent
to an echo server will be copied directly back to the client. The exact details of implementing
an echo server might vary depending on what kind of I/O framework you are using, but it's
inevitable that somewhere in the heart of an echo server there is some kind of data structure,
either implicit or explicit, serving as a byte queue or circular buffer. If you define an
abstract interface around this data structure, you have something that captures the admittedly
trivial semantics of the echo protocol, but is a pure data structure, and therefore doesn't
need to interact with the OS at all. We now have the echo engine depicted below. A simple
circular buffer or byte queue, but with a slightly fancier title. With this abstract interface,
however, we can start tweaking the implementation of our engine independently from the rest
of our server.</p>



<map name='GLIFFY_MAP_30752712_Echo_Engine'></map>
<table width="100%">
    <tr>
        <td align="center">
            <table>
                <caption align="bottom">
                                    </caption>
                <tr>
                    <td>
                        <img style="border: none; width: 840px; height: 214px;"
                                                          usemap="#GLIFFY_MAP_30752712_Echo_Engine"
                                                          src="/confluence/download/attachments/30752712/Echo+Engine.png?version=2&amp;modificationDate=1360509983000"
                             alt="A&amp;#32;Gliffy&amp;#32;Diagram&amp;#32;named&amp;#58;&amp;#32;Echo&amp;#32;Engine"/>
                    </td>
                </tr>
            </table>
        </td>
    </tr>
</table>




<p>Let's start by making our example do something a little more interesting. We can
add a query interface to our engine as depicted below. The I/O portion of our application
still views the engine as an ordinary circular buffer, however we can now add application
logic that uses the query interface to take action. The exact nature of the query interface
depends on the protocol and on the engine internals. A simplistic query interface might just
provide a byte count of data passing through, whereas a more sophisticated engine could decode
and make available semantic content embedded within the byte stream. While a little bit more
capable than a simple echo engine, this kind of engine is still just a simple data structure
with some added interrogatives, and so retains the desired decoupling and portability.</p>



<map name='GLIFFY_MAP_30752712_Echo_Engine_with_Query_Interface'></map>
<table width="100%">
    <tr>
        <td align="center">
            <table>
                <caption align="bottom">
                                    </caption>
                <tr>
                    <td>
                        <img style="border: none; width: 840px; height: 306px;"
                                                          usemap="#GLIFFY_MAP_30752712_Echo_Engine_with_Query_Interface"
                                                          src="/confluence/download/attachments/30752712/Echo+Engine+with+Query+Interface.png?version=2&amp;modificationDate=1360510144000"
                             alt="A&amp;#32;Gliffy&amp;#32;Diagram&amp;#32;named&amp;#58;&amp;#32;Echo&amp;#32;Engine&amp;#32;with&amp;#32;Query&amp;#32;Interface"/>
                    </td>
                </tr>
            </table>
        </td>
    </tr>
</table>




<p>Now consider modifying our example to be a simple request/response protocol where
every response is a predefined function of each request. We can provide an engine for this
kind of protocol by modifying our simple echo engine to detect requests and replace them with
responses. With the transformation function entirely encapsulated inside the engine, the interface
presented is completely identical to our original echo engine. However, unlike the echo engine,
we now have the ability to capture a larger class of protocols.</p>



<map name='GLIFFY_MAP_30752712_Simple_Request-Response_Engine'></map>
<table width="100%">
    <tr>
        <td align="center">
            <table>
                <caption align="bottom">
                                    </caption>
                <tr>
                    <td>
                        <img style="border: none; width: 840px; height: 214px;"
                                                          usemap="#GLIFFY_MAP_30752712_Simple_Request-Response_Engine"
                                                          src="/confluence/download/attachments/30752712/Simple+Request-Response+Engine.png?version=2&amp;modificationDate=1360510219000"
                             alt="A&amp;#32;Gliffy&amp;#32;Diagram&amp;#32;named&amp;#58;&amp;#32;Simple&amp;#32;Request&amp;#45;Response&amp;#32;Engine"/>
                    </td>
                </tr>
            </table>
        </td>
    </tr>
</table>




<p>Now just as we did with the echo server, we can also extend the simple request/response
scenario with a query interface as depicted below. And as before, our engine is still just
a data structure with well defined interfaces that keep it decoupled and portable.</p>



<map name='GLIFFY_MAP_30752712_Simple_Request-Response_Engine_with_Query_Interfac'></map>
<table width="100%">
    <tr>
        <td align="center">
            <table>
                <caption align="bottom">
                                    </caption>
                <tr>
                    <td>
                        <img style="border: none; width: 839px; height: 313px;"
                                                          usemap="#GLIFFY_MAP_30752712_Simple_Request-Response_Engine_with_Query_Interfac"
                                                          src="/confluence/download/attachments/30752712/Simple+Request-Response+Engine+with+Query+Interfac.png?version=2&amp;modificationDate=1360510311000"
                             alt="A&amp;#32;Gliffy&amp;#32;Diagram&amp;#32;named&amp;#58;&amp;#32;Simple&amp;#32;Request&amp;#45;Response&amp;#32;Engine&amp;#32;with&amp;#32;Query&amp;#32;Interfac"/>
                    </td>
                </tr>
            </table>
        </td>
    </tr>
</table>




<p>Despite being able to capture a large class of protocols, the simple request/response
engine still has some significant limitations. For example if the response transformation
requires accessing a file, as is the case with a classic HTTP implementation, our engine is
no longer decoupled from the OS, and could in fact block on file I/O. Also, if the request
is actually invoking application logic such as with an RPC protocol or with most modern usage
of HTTP, we simply can't pre-define the result within the confines of our library. To accommodate
these scenarios, we need to modify our engine a little further to allow the application to
control aspects of the response. To do this we add a control interface that can be used in
conjunction with the query interface to inspect a request and modify the response.</p>

<p>As we do this we begin to see two distinct facets of the engine emerge. We have the
interface presented to the I/O portion of our server which remains identical to our original
echo engine, i.e. a simple circular buffer. Independent of this we have a higher level API
emerging that presents the semantics of the protocol to the part of the server dealing with
application logic. As we've seen, the latter interface can evolve and change depending on
the nature of the protocol without any impact on the former. The I/O portion of our server
need not know what kind of processing happens to the byte stream as it passes through the
engine.</p>



<map name='GLIFFY_MAP_30752712_Request-Response_Engine'></map>
<table width="100%">
    <tr>
        <td align="center">
            <table>
                <caption align="bottom">
                                    </caption>
                <tr>
                    <td>
                        <img style="border: none; width: 841px; height: 315px;"
                                                          usemap="#GLIFFY_MAP_30752712_Request-Response_Engine"
                                                          src="/confluence/download/attachments/30752712/Request-Response+Engine.png?version=3&amp;modificationDate=1360510817000"
                             alt="A&amp;#32;Gliffy&amp;#32;Diagram&amp;#32;named&amp;#58;&amp;#32;Request&amp;#45;Response&amp;#32;Engine"/>
                    </td>
                </tr>
            </table>
        </td>
    </tr>
</table>




<p>While the above design nicely captures a very general class of protocol, it is still
limited to protocols that use only request/response style interactions. The output bytes are
always a function of the input bytes, even though the nature of that function may be influenced
by the control interface. The final step to a fully general purpose protocol engine is to
define the output bytes as a function of both the input bytes <b>and</b> the control
interface as depicted below. This allows for fully general protocols whose interactions are
not so regimented as request/response.</p>

<p>It is important to note with the general purpose engine depicted below, while the
I/O portion of the application is almost identical to our original echo server, there is one
important difference. For the echo protocol pushing bytes into the engine is the only thing
that will cause them to be produced, whereas for the general purpose engine, any access to
the control interface may cause bytes to be produced. </p>



<map name='GLIFFY_MAP_30752712_General_Engine'></map>
<table width="100%">
    <tr>
        <td align="center">
            <table>
                <caption align="bottom">
                                    </caption>
                <tr>
                    <td>
                        <img style="border: none; width: 840px; height: 322px;"
                                                          usemap="#GLIFFY_MAP_30752712_General_Engine"
                                                          src="/confluence/download/attachments/30752712/General+Engine.png?version=3&amp;modificationDate=1360510994000"
                             alt="A&amp;#32;Gliffy&amp;#32;Diagram&amp;#32;named&amp;#58;&amp;#32;General&amp;#32;Engine"/>
                    </td>
                </tr>
            </table>
        </td>
    </tr>
</table>




<p>A further refinement of the general purpose engine is to separate the control interface
into a distinct entity with its own lifecycle. This entity can be bound to a given transport
and then unbound. This separation formalizes the two facets of the engine interface discussed
above into a bottom half that can keep a very simple I/O oriented view of the world, and a
top half that reflects the high level application interface to the protocol. The distinct
lifecycle also permits the control interface to carry state across multiple physical connections,
which can allow for a much more sophisticated control entity that can provide a simpler interface,
for example automatically recovering state from a previous connection as is done with HTTP
sessions.</p>



<map name='GLIFFY_MAP_30752712_Seperate_Control_Interface_Lifcycle'></map>
<table width="100%">
    <tr>
        <td align="center">
            <table>
                <caption align="bottom">
                                    </caption>
                <tr>
                    <td>
                        <img style="border: none; width: 840px; height: 513px;"
                                                          usemap="#GLIFFY_MAP_30752712_Seperate_Control_Interface_Lifcycle"
                                                          src="/confluence/download/attachments/30752712/Seperate+Control+Interface+Lifcycle.png?version=2&amp;modificationDate=1360511259000"
                             alt="A&amp;#32;Gliffy&amp;#32;Diagram&amp;#32;named&amp;#58;&amp;#32;Seperate&amp;#32;Control&amp;#32;Interface&amp;#32;Lifcycle"/>
                    </td>
                </tr>
            </table>
        </td>
    </tr>
</table>



    </div>
        <div id="commentsSection" class="wiki-content pageSection">
        <div style="float: right;">
            <a href="https://cwiki.apache.org/confluence/users/viewnotifications.action"
class="grey">Change Notification Preferences</a>
        </div>
        <a href="https://cwiki.apache.org/confluence/display/qpid/Protocol+Engines">View
Online</a>
        |
        <a href="https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=30752712&revisedVersion=34&originalVersion=33">View
Changes</a>
                |
        <a href="https://cwiki.apache.org/confluence/display/qpid/Protocol+Engines?showComments=true&amp;showCommentArea=true#addcomment">Add
Comment</a>
            </div>
</div>
</div>
</div>
</div>
</body>
</html>

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@qpid.apache.org
For additional commands, e-mail: commits-help@qpid.apache.org


Mime
View raw message