Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CAF_KkPyE24bKs80XBWpWutMjT-wXZrx7r5KDD+OQr+aQXhL2sA@mail.gmail.com>
References: 
 <CAF_KkPyE24bKs80XBWpWutMjT-wXZrx7r5KDD+OQr+aQXhL2sA@mail.gmail.com>
Date: Tue, 2 Oct 2012 14:24:32 -0400
Message-ID: 
 <CAGUtCHrrnoiGtDO=v_T=jV6OeyNp77CF_wU6j5xK9ZEQQ1HbSg@mail.gmail.com>
Subject: Re: compressing values returned to scanner
From: Keith Turner <keith@deenlo.com>
To: user@accumulo.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Oct 1, 2012 at 3:03 PM, ameet kini <ameetkini@gmail.com> wrote:
>
> My understanding of compression in Accumulo 1.4.1 is that it is on by
> default and that data is decompressed by the tablet server, so data on the
> wire between server/client is decompressed. Is there a way to shift the
> decompression from happening on the server to the client? I have a use case
> where each Value in my table is relatively large (~ 8MB) and I can benefit
> from compression over the wire. I don't have any server side iterators, so
> the values don't need to be decompressed by the tablet server. Also, each
> scan returns a few rows, so client-side decompression can be fast.
>
> The only way I can think of now is to disable compression on that table, and
> handle compression/decompression in the application. But if there is a way
> to do this in Accumulo, I'd prefer that.
>

There are two levels of compression in Accumulo.  First redundant
parts of the key are not stored.  If the row in a key is the same as
the previous row, then its not stored again.   The same is done for
columns and time stamps.   After the relative encoding is done a block
of key values is then compressed with gzip.

As data is read from an RFile, when the row of a key is the same as
the previous key it will just point to the previous keys row.  This is
carried forward over the wire.  As keys are transferred, duplicate
fields in the key are not transferred.

As far as decompressing on the client side vs server side, the server
at least needs to decompress keys.  On the server side you usually
need to read from multiple sorted files and order the result.  So you
need to decompress  keys on the server side to compare them.  Also
iterators on the server side need the keys and values decompressed.

> Thanks,
> Ameet