Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D826DFA3B for ; Tue, 19 Mar 2013 16:31:39 +0000 (UTC) Received: (qmail 70402 invoked by uid 500); 19 Mar 2013 16:31:37 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 70378 invoked by uid 500); 19 Mar 2013 16:31:37 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 70370 invoked by uid 99); 19 Mar 2013 16:31:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Mar 2013 16:31:37 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [63.146.121.108] (HELO mail.venarc.com) (63.146.121.108) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Mar 2013 16:31:30 +0000 Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.venarc.com (Postfix) with ESMTP id EF6556F00001 for ; Tue, 19 Mar 2013 09:31:08 -0700 (PDT) X-Virus-Scanned: amavisd-new at venarc.com Received: from mail.venarc.com ([127.0.0.1]) by localhost (mail.venarc.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id W4DfNc43xvZB for ; Tue, 19 Mar 2013 09:31:08 -0700 (PDT) Received: from [192.168.1.2] (drew-home [108.60.62.58]) by mail.venarc.com (Postfix) with ESMTPSA id 2B7BF6F00002 for ; Tue, 19 Mar 2013 09:31:08 -0700 (PDT) From: Drew Kutcharian Content-Type: multipart/alternative; boundary="Apple-Mail=_CA32CBA6-1178-4044-B05F-BAE94220FC96" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Cassandra Compression and Wide Rows Date: Tue, 19 Mar 2013 09:31:06 -0700 References: <36F3BB33-4A95-4EC9-82E1-1022B391958B@venarc.com> <425BB682-26C7-4039-A773-7C39882E685F@venarc.com> <83073270-80EA-4394-BF11-B91919FFC953@venarc.com> To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_CA32CBA6-1178-4044-B05F-BAE94220FC96 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 Thanks Sylvain. So C* compression is block based and has nothing to do = with format of the rows. On Mar 19, 2013, at 1:31 AM, Sylvain Lebresne = wrote: > That's just describing what compression is about. Compression (not in = C*, in general) is based on recognizing repeated pattern. >=20 > So yes, in that sense, static column families are more likely to yield = better compression ratio because it is more likely to have repeated = patterns in the compressed blocks. But: > 1) it doesn't necessarily mean that wide column families won't have a = good compression ratio per se. > 2) you can absolutely have crappy compression ratio with a static = column family. Just create a column family where each row has 1 column = 'image' that contains a png. >=20 > And to come back to your initial question, I highly doubt disk level = compression would be much of a workaround because again, that's more = about how compression is working than how Cassandra use it. >=20 > At the end of the day, I really think the best choice is to try it and = decide for yourself if it does more good than harm or the converse. >=20 > -- > Sylvain =20 >=20 >=20 > On Tue, Mar 19, 2013 at 3:58 AM, Drew Kutcharian = wrote: > Edward/Sylvain, >=20 > I also came across this post on DataStax's blog: >=20 >> When to use compression >> Compression is best suited for ColumnFamilies where there are many = rows, with each row having the same columns, or at least many columns in = common. For example, a ColumnFamily containing user data such as = username, email, etc., would be a good candidate for compression. The = more similar the data across rows, the greater the compression ratio = will be, and the larger the gain in read performance. >> Compression is not as good a fit for ColumnFamilies where each row = has a different set of columns, or where there are just a few very wide = rows. Dynamic column families such as this will not yield good = compression ratios. >=20 > = http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-compression >=20 > @Sylvain, does this still apply on more recent versions of C*? >=20 >=20 > -- Drew >=20 >=20 >=20 > On Mar 18, 2013, at 7:16 PM, Edward Capriolo = wrote: >=20 >> I feel this has come up before. I believe the compression is block = based, so just because no two column names are the same does not mean = the compression will not be effective. Possibly in their case the = compression was not effective. >>=20 >> On Mon, Mar 18, 2013 at 9:08 PM, Drew Kutcharian = wrote: >> That's what I originally thought but the OOYALA presentation from = C*2012 got me confused. Do you guys know what's going on here? >>=20 >> The video: = http://www.youtube.com/watch?v=3Dr2nGBUuvVmc&feature=3Dplayer_detailpage#t= =3D790s >> The slides: Slide 22 @ = http://www.datastax.com/wp-content/uploads/2012/08/C2012-Hastur-NoahGibbs.= pdf >>=20 >> -- Drew >>=20 >>=20 >> On Mar 18, 2013, at 6:14 AM, Edward Capriolo = wrote: >>=20 >>>=20 >>> Imho it is probably more efficient for wide. When you decompress 8k = blocks to get at a 200 byte row you create overhead , particularly young = gen. >>> On Monday, March 18, 2013, Sylvain Lebresne = wrote: >>> > The way compression is implemented, it is oblivious to the CF = being wide-row or narrow-row. There is nothing intrinsically less = efficient in the compression for wide-rows. >>> > -- >>> > Sylvain >>> > >>> > On Fri, Mar 15, 2013 at 11:53 PM, Drew Kutcharian = wrote: >>> >> >>> >> Hey Guys, >>> >> >>> >> I remember reading somewhere that C* compression is not very = effective when most of the CFs are in wide-row format and some folks = turn the compression off and use disk level compression as a workaround. = Considering that wide rows with composites are "first class citizens" in = CQL3, is this still the case? Has there been any improvements on this? >>> >> >>> >> Thanks, >>> >> >>> >> Drew >>> > >>=20 >>=20 >=20 >=20 --Apple-Mail=_CA32CBA6-1178-4044-B05F-BAE94220FC96 Content-Transfer-Encoding: 7bit Content-Type: text/html; charset=iso-8859-1 Thanks Sylvain. So C* compression is block based and has nothing to do with format of the rows.

On Mar 19, 2013, at 1:31 AM, Sylvain Lebresne <sylvain@datastax.com> wrote:

That's just describing what compression is about. Compression (not in C*, in general) is based on recognizing repeated pattern.

So yes, in that sense, static column families are more likely to yield better compression ratio because it is more likely to have repeated patterns in the compressed blocks. But:
1) it doesn't necessarily mean that wide column families won't have a good compression ratio per se.
2) you can absolutely have crappy compression ratio with a static column family. Just create a column family where each row has 1 column 'image' that contains a png.

And to come back to your initial question, I highly doubt disk level compression would be much of a workaround because again, that's more about how compression is working than how Cassandra use it.

At the end of the day, I really think the best choice is to try it and decide for yourself if it does more good than harm or the converse.

--
Sylvain  


On Tue, Mar 19, 2013 at 3:58 AM, Drew Kutcharian <drew@venarc.com> wrote:
Edward/Sylvain,

I also came across this post on DataStax's blog:

When to use compression
Compression is best suited for ColumnFamilies where there are many rows, with each row having the same columns, or at least many columns in common. For example, a ColumnFamily containing user data such as username, email, etc., would be a good candidate for compression. The more similar the data across rows, the greater the compression ratio will be, and the larger the gain in read performance.
Compression is not as good a fit for ColumnFamilies where each row has a different set of columns, or where there are just a few very wide rows. Dynamic column families such as this will not yield good compression ratios.


@Sylvain, does this still apply on more recent versions of C*?


-- Drew



On Mar 18, 2013, at 7:16 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote:

I feel this has come up before. I believe the compression is block based, so just because no two column names are the same does not mean the compression will not be effective. Possibly in their case the compression was not effective.

On Mon, Mar 18, 2013 at 9:08 PM, Drew Kutcharian <drew@venarc.com> wrote:
That's what I originally thought but the OOYALA presentation from C*2012 got me confused. Do you guys know what's going on here?


-- Drew


On Mar 18, 2013, at 6:14 AM, Edward Capriolo <edlinuxguru@gmail.com> wrote:


Imho it is probably more efficient for wide. When you decompress 8k blocks to get at a 200 byte row you create overhead , particularly young gen.
On Monday, March 18, 2013, Sylvain Lebresne <sylvain@datastax.com> wrote:
> The way compression is implemented, it is oblivious to the CF being wide-row or narrow-row. There is nothing intrinsically less efficient in the compression for wide-rows.
> --
> Sylvain
>
> On Fri, Mar 15, 2013 at 11:53 PM, Drew Kutcharian <drew@venarc.com> wrote:
>>
>> Hey Guys,
>>
>> I remember reading somewhere that C* compression is not very effective when most of the CFs are in wide-row format and some folks turn the compression off and use disk level compression as a workaround. Considering that wide rows with composites are "first class citizens" in CQL3, is this still the case? Has there been any improvements on this?
>>
>> Thanks,
>>
>> Drew
>





--Apple-Mail=_CA32CBA6-1178-4044-B05F-BAE94220FC96--