cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Hood <>
Subject Re: CQL3 Frame Length
Date Thu, 24 Jan 2013 10:55:57 GMT
This may be stretching this thread a little bit too far, but I think SCTP would lend itself
nicely to the CQL3 execution model (as an alternative to TCP). You would get message framing
and asynchronous delivery for free, so you could just write each CQL message in it's entirety
without having to precompute the length. On the receiving end, the SCTP layer would feed your
library with each message as an when they occur. All of the looking ahead, parallel decoding
and processing and related head of line concerns would be implemented by the carrying protocol
rather than in CQL3. The downside is that you'd need to write an SCTP based client.

Also, despite potentially being a good idea, it is questionable as to whether SCTP is ever
going to get adopted in any case. I'm just mentioning it because it seems to be quite a nice
fit for the problem at hand.  

On Monday, 21 January 2013 at 23:05, Pierre Chalamet wrote:

> Hi,
> That’s not a good reason imho. This would have been better to have chunks of data (like
in the good old IFF file format).
> If the client is not able to read the chunk, just skip it. And frankly, that’s not
a few more bytes that would have killed us.
> As an example, request tracing was added pretty late and then, additional data just landed
not at the end as could have been anticipated, but before the body of the frame. This could
have been handled transparently with a chunk format. And OK this was in rc2 and not in 1.2
so no regression officially introduced.  
> But well, it’s v1 – there are still 0x7E more versions to get it better.
> - Pierre
> From: Theo Hultberg []  
> Sent: Saturday, January 19, 2013 6:33 PM
> To: (
> Subject: Re: CQL3 Frame Length
> Hi,
> Another reason for keeping the frame length in the header is that newer versions can
add fields to frames without older clients breaking. For example a minor release can add some
more content to an existing frame without older clients breaking. If clients didn't know the
full frame length (and were required by the specification to consume all the bytes) there
would be trailing garbage which would most likely crash the client.
> T#
> > Hey Sylvain,>
> > Thanks for explaining the rationale. When you look at from the perspective
> > of the use cases you mention, it makes sense to be able to supply the
> > reader with the frame size up front.>
> > I've opted to go for serializing the frame into a buffer. Although this
> > could materialize an arbitrarily large amount of memory, ultimately the
> > driving application has control of the degree to which this can occur, so
> > in the grander scheme of things, you can still maintain streaming semantics.>
> > Thanks for the heads up.>
> > Cheers,>
> > Ben>
> >
> > On Tue, Jan 8, 2013 at 4:08 PM, Sylvain Lebresne < (>wrote:>
> >> Mostly this is because having the frame length is convenient to have in
> >> practice.
> >>
> >> Without pretending that there is only one way to write a server, it is
> >> common
> >> to separate the phase "read a frame from the network" from the phase
> >> "decode
> >> the frame" which is often simpler if you can read the frame upfront. Also,
> >> if
> >> you don't have the frame size, it means you need to decode the whole frame
> >> before being able to decode the next one, and so you can't parallelize the
> >> decoding.
> >>
> >> It is true however that it means for the write side that you need to
> >> either be
> >> able to either pre-compute the frame body size or to serialize it in memory
> >> first. That's a trade of for making it easier on the read side. But if you
> >> want
> >> my opinion, on the write side too it's probably worth parallelizing the
> >> message
> >> encoding (which require you encode it in memory first) since it's an
> >> asynchronous protocol and so there will likely be multiple writer
> >> simultaneously.
> >>
> >> --
> >> Sylvain
> >>
> >>
> >>
> >> On Tue, Jan 8, 2013 at 12:48 PM, Ben Hood < (>
> >>
> >>> Hi,
> >>>
> >>> I've read the CQL wire specification and naively, I can't see how the
> >>> frame length length header is used.
> >>>
> >>> To me, it looks like on the read side, you know which type of structures
> >>> to expect based on the opcode and each structure is TLV encoded.
> >>>
> >>> On the write side, you need to encode TLV structures as well, but you
> >>> don't know the overall frame length until you've encoded it. So it would
> >>> seem that you either need to pre-calculate the cumulative TLV size before
> >>> you serialize the frame body, or you serialize the frame body to a buffer
> >>> which you can then get the size of and then write to the socket, after
> >>> having first written the count out.
> >>>
> >>> Is there potentially an implicit assumption that the reader will want to
> >>> pre-buffer the entire frame before decoding it?
> >>>
> >>> Cheers,
> >>>
> >>> Ben
> >>>
> >>
> >>  

View raw message