geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hitesh Khamesra <>
Subject Re: [gemfire-dev] New Client-Server Protocol Proposal
Date Fri, 05 May 2017 21:09:48 GMT

0. In first phase we are not doing chunking/fragmentation. And even this will be option for
1. Are you refereeing websocket/spdy? But I think we are talking almost same thing, may be
push isPartialMessage flag with chunk length(Anthony's example below) ?
2. That's the part of the problem. Even if you need to serialize the "String", you need to
write length first and then need to write serialized utf bytes. We can implement chunked input
stream and can de-serialize the object as it is coming (DataSerializable.fromData(ChunkedStream)).

      From: Jacob Barrett <>
 To:; Hitesh Khamesra <> 
Cc: Anthony Baker <>
 Sent: Friday, May 5, 2017 7:29 AM
 Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal

On Thu, May 4, 2017 at 2:52 PM Hitesh Khamesra <> wrote:

Basically, thread/layer should not hold any resources while serializing the object or chunk. 
We should be able to see this flow (ms1-chunk1, msg2-chunk1, ms1-chunk2, msg3-chunk, msg2-chunk2,
so on ...)

Correct, but putting that in the message layer is not appropriate. The simple solution is
that the multiple channels can be achieved with multiple sockets. The later optimization is
to add a channel multiplexer layer between the message and socket layers. 
If we put it in the message layer, not only does it for the message to tackle something it
shouldn't be concerned with, reassembling itself, but it also forces all implementors to tackle
this logic up front. By layering we can release without, implementors aren't forced into understanding
the logic, and later we can release the layers and the client can negotiate.
On other pdx note: to de-serialize the pdx we need length of serialized bytes, so that we
can read field offset from serialized stream, and then can read field value. Though, I can
imagine with the help of pdxType, we can interpret serialized stream.

Yes, so today PDX serialization would be no worse, the PDX serializer would have to buffer,
but other may not have to. The length of the buffered PDX could be used as the first chunk
length and complete in single chunk. Although, I suspect that amortized overhead of splitting
the chunks  will be nil anyway. 
The point is that the message encoding of values should NOT have any unbounded length fields
and require long or many buffers to complete serialization. By chunking you can accomplish
this by not needing to buffer the whole stream, just small (say 1k), chunks at a time to get
the chunk length. 
Buffers == Latency

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message