hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lars hofhansl <lhofha...@yahoo.com>
Subject RPC KeyValue encoding
Date Sun, 02 Sep 2012 04:03:44 GMT
In 0.96 we changing the wire protocol to use protobufs.

While we're at it, I am wondering whether we can optimize a few things:

1. A Put or Delete can send many KeyValues, all of which have the same row key and many will
likely have the same column family.
2. Likewise a Scan result or Get is for a single row. Each KV will again will have the same
row key and many will have the same column family.
3. The client and server do not need to share the same KV implementation as long as they are
(de)serialized the same. KVs on the server will be backed by a shared larger byte[] (the block
reads from disk), the KVs in the memstore will probably have the same implementation (to use
slab, but maybe even here it would be benificial to store the row key and CF separately and
share between KV where possible). Client KVs on the other hand could share a row key and or
column family.

This would require a KeyValue interface and two different implementations; one backed by a
byte[] another that stores the pieces separately. Once that is done one could even envision
KVs backed by a byte buffer.

Both (de)serialize the same, so when the server serializes the KVs it would send the row key
first, then the CF, then column, TS, finally followed by the value. The client could deserialize
this and directly reuse the shared part in its KV implementation.
That has the potentially to siginificantly cut down client/server network IO and save memory
on the client, especially with wide columns.

Turning KV into an interface is a major undertaking. Would it be worth the effort? Or maybe
the RPC should just be compressed?

We'd have to do that before 0.96.0 (I think), because even protobuf would not provide enough
flexibility to make such a change later - which incidentally leads to another discussion about
whether client and server should do an initial handshake to detect each others version, but
that is a different story.

-- Lars

View raw message