cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Schuller <peter.schul...@infidyne.com>
Subject Re: Data model design question
Date Sat, 20 Nov 2010 09:21:42 GMT
>  Our team decided to use Cassandra as storage solution to a dataset.
> I am very new to the NoSQL world and Cassandra so I am hoping to get
> some help from the community: The dataset is pretty simple, we have
> for each key a number of columns with values. Each day we compute a
> new version of this dataset, the new version will mostly update
> existing keys but could also add and delete some keys. (And we'll
> build a service that queries Cassandra). A key requirement for us is
> we want to keep versions of the dataset and keep N versions around,
> this is in case we discover problems in the current version and need
> to "roll up" to an older one. I thought about creating a Column Family
> per version, this means we will create a new column family every day
> and occasionally delete column families according to some truncation
> policy. I know Cassandra 0.7 now makes changing schema easier, but is
> this a good way to go? I would really like to hear what you guys think
> is the better way to handle this. Thank you.

What you propose would presumably work and in some sense be efficient
if you are intending to re-load the entire data set each day. However,
I wouldn't say column families are intended to be used like this and
it sounds like it would be better to model the data differently.
Column families have implications on e.g. memtables sizes; they are a
bit sensitive in the way you submit them to a cluster (only one schema
change may be propagating at a time, etc). In addition it implies that
you have to keep track of and manage the schema somehow at a
meta-level.

While I don't see an actual unavoidable problem with it (e.g. memtable
sizes is not an issue if you completely stop writing to old memtables,
schema change propagation can of course be solved, etc), my feeling is
that you are likely better off modelling your data differently.

It sounds like a possible way to go  would be a CF with super columns
whose names are your time periods and whose values are your columns.
That assumes a small amount of data per row in each period. This way
you will have access to the latest version of your data by doing a
slice of size 1 (sorting on the period), while older versions will be
remain accessible and individually delete:able. This assumes each
super column is updated completely or not at all, and that you're not
having to do merges across multiple super columns in your application.

Something to keep in mind here is that all insertions of new data and
updates will be to pre-existing rows (unless the row key is completely
new) so you would expect reads to be spread out over sstables on disk.

An alternative is to introduce a level of indirection (if you can live
with that); have a column family that maps rows to columns whose names
are your periods and whose values are row keys for another column
family. Then each row in the pointed-to column family will be your
data for that particular period. This would probably be advantageous
in particular if you intend to have a lot of data associated with each
key, since the data would directly map to a single row in Cassandra
(and large rows are okay and give you efficient access to individual
columns or column ranges).

(But it still assumes you're updating either all the data or no data
for a given row key in a particular time period.)

-- 
/ Peter Schuller

Mime
View raw message