geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hitesh Khamesra <>
Subject Re: [gemfire-dev] New Client-Server Protocol Proposal
Date Thu, 04 May 2017 17:38:53 GMT
Hi Anthony:

Help me to understand data chunking here?

>> bytes => arbitrary byte[] that can be chunked

Message => MessageHeader MessageBody

So lets say we want to send long byte[] into two chunks, then we will send two messages? And
other side will combine those two messages using "correlationId" ?


From: Anthony Baker <>
Cc: Hitesh Khamesra <>
Sent: Wednesday, May 3, 2017 5:42 PM
Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal

> On May 3, 2017, at 1:33 PM, Galen M O'Sullivan <> wrote:
> On Tue, May 2, 2017 at 11:52 AM, Hitesh Khamesra <
>> wrote:
>> Absolutely its a implementation detail.
> This doesn't answer Dan's comment. Do you think fragmentation should be
> taken care of by the TCP layer or the protocol should deal with it
> specifically?

There’s some really good feedback and discussion in this thread!  Here are a few thoughts:

1) Optional metadata should be used for fields that are generally applicable across all messages.
 If a metadata field is required or only applies to a small set of messages, it should become
part of a message definition.  Of course there’s some grey area here.

2) I think we should pull out the message fragmentation support to avoid some significant
complexity.  We can later add a fragmentation / envelope layer on top without disrupting the
current proposal.  I do think we should add the capability for chunking data (more on that

3) I did not find any discussion of message pipelining (queuing multiple requests on a socket
without waiting for a response) or out-of-order responses.  What is the plan for these capabilities
and how will that affect consistency?  What about retries?

4) Following is an alternative definition with these characteristics:

- Serialized data can either be primitive or encoded values.  Encoded values are chunked as
needed to break up large objects into a series of smaller parts.
- Because values can be chunked, the size field is removed.  This allows the message to be
streamed to the socket incrementally.
- The apiVersion is removed because we can just define a new body type with a new apiId (e.g.
GetRequest2 with aipId = 1292).
- The GetRequest tells the server what kind of encoding the client is able to understand.
- The metadata map is not used for fields that belong in the message body.  I think it’s
much easier to write a spec without if statements :-)

Message => MessageHeader MessageBody

MessageHeader => correlationId metadata
    correlationId => integer
    metadata => count (key value)*
        count => integer
        key => string
        value => string

MessageBody => apiId body
    apiId => integer
    body => (see specific definitions)

GetRequest => 0 acceptEncoding key
    0 => the API id
    acceptEncoding => (define some encodings for byte[], JSON, PDX, *, etc)
    key => EncodedValue

GetResponse => 1 value
    1 => the API id
    value => EncodedValue

PutRequest => 2 eventId key value
    2 => the API id
    eventId => clientId threadId sequenceId
        clientId => string
        threadId => integer
        sequenceId => integer
    key => EncodedValue
    value => EncodedValue

EncodedValue => encoding (boolean | integer | number | string | ((length bytes)* 0))
    encoding => (define some encodings for byte[], JSON, PDX, *, etc)
    boolean => TRUE or FALSE
    integer => a signed integer value
    number => a decimal value corresponding to IEEE 754
    string => UTF-8 text
    bytes => arbitrary byte[] that can be chunked


View raw message