Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of rajkumar.w93@gmail.com
 designates 209.85.161.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=LqrdysSPOq3bAl8J123QzsYb0GikB74csT6AWTBtRqjA/Jv+UheMXHwUw5fLFWSe7c
         Mor2R4PoMStNZa4Qh4jlUmnnFwO4Km9ggZPhKkIyk5tUQ+vlGmHcLKQPdEka/ou4AI2a
         6WwUMSHLZ0tEe8auDuKfB33LZ5l+/nuzWXANY=
MIME-Version: 1.0
Sender: rajkumar.w93@gmail.com
In-Reply-To: <AANLkTi=3_8N-muWOUib=esuc9aEZiobLE46P4jwzhA7_@mail.gmail.com>
References: <AANLkTimMp3nSuPdAW-OhWoVXbmNd-Dai+0DC25GH1NVM@mail.gmail.com>
	<AANLkTi=2j_x5RauO47W7HBzBQ1zu=-WxmKE92PC-fKL-@mail.gmail.com>
	<AANLkTikBazEBkPi6UQzjXkx_vX+=0Y8PkpgymkkzWdJ_@mail.gmail.com>
	<AANLkTimS4O86hpdBe5TEcEWEEtj+0TuwkJqYx8=7pZ5_@mail.gmail.com>
	<AANLkTi=3_8N-muWOUib=esuc9aEZiobLE46P4jwzhA7_@mail.gmail.com>
Date: Sun, 6 Feb 2011 09:56:28 +0530
Message-ID: <AANLkTi=Fk9KL_+Oe-T=TZTUOFW5v+AR-5XYhTCMGq20E@mail.gmail.com>
Subject: Re: Merging the rows of two column families(with similar attributes)
 into one ??
From: Ertio Lew <ertiop93@gmail.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks Tyler!

I think I'll have to very carefully take into consideration all these
factors before deciding upon how to split my data into CFs, as this
cannot an objective answer. I am expecting around atleast 8 column
families for my entire application, if I split the data strictly
according to the various features and requirements of the application.

I think there should have been provision for specifying on per query
basis, what rows be cached while you're reading them, from a row_cache
enabled CF. Thus you could easily merge similar data for different
features of your application in a single CF. I believe, this would
have also lead to much more efficient use of the cache space!!( if you
were using same data for different parts in your app which have
different caching needs)

Regards,

Ertio

On Sun, Feb 6, 2011 at 1:22 AM, Tyler Hobbs <tyler@datastax.com> wrote:
>> if you have under control parameters like
>> memtable_throughput & memtable_operations which are set per column
>> family basis then you can directly control & adjust by splitting the
>> memory space between two CFs in proportion to what you would do in
>> single CF.
>> Hence there should be no extra memory consumption for multiple CFs
>> that have been split from single one??
>
> Yes, I think you have the right idea here.=A0 This is a small amount of
> overhead for the extra memtable and keeping track of a second set of
> indexes, bloom filters, sstables, etc.
>
>> Regarding the compactions, I think even if they are more the size of
>> the SST files to be compacted is smaller as the data has been split
>> into two.
>> Then more compactions but smaller too!!
>
> Yes.
>
>> if some CF is written less often as compared to other CFs, then the
>> memtable would consume space in the memory until it is flushed, this
>> memory space could have been much better used by a CF that's heavily
>> written and read. And if you try to make the thresholds for flush
>> smaller then more compactions would be needed.
>
> If you merge the two CFs together, then updates to the 'less freqent' row=
s
> will still consume memory, only it will all be within one memtable.
> (Memtables grow in size until they are flushed, they don't reserve some s=
et
> amount of memory.)=A0 Furthermore, because your memtables will be filled =
up by
> the 'more frequent' rows, the 'less frequent' rows will get fewer
> updates/overwrites in memory, so they will tend to be spread across a
> greater number of SSTables.
>
> --
> Tyler Hobbs
> Software Engineer, DataStax
> Maintainer of the pycassa Cassandra Python client library
>
>