cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Turner <>
Subject Re: Cassandra data model help
Date Thu, 09 Aug 2012 17:37:20 GMT
On Thu, Aug 9, 2012 at 5:52 AM,  <> wrote:
> Hi,
> I am trying to create a Cassandra schema for cluster monitoring system, where one cluster
can have multiple nodes and I am monitoring multiple matrices from a node. My raw data schema
looks like and taking values in every 5 min interval
> matrix_name + daily time stamp as row key, composite column name of node name and time
stamp and matrix value as column value
> the problem I am facing is a node can go back and forth between the clusters(system can
have more than one clusters) so if i need monthly statistics plotting of a cluster I have
to consider the nodes that are leaving and joining during this period of time, some node might
be part of the cluster for just 15 days and some could join the cluster last 10 day of month,
so to plot data for a particular cluster for a time interval I need to know the nodes which
were part of that cluster for that period of time, what could be the best schema for this
solution ? I have tried few ideas so far no luck, any suggestions ?

Store each node stat in it's own row.  Then decide if you want to
track when a node joins/leaves a cluster so you can build the aggs on
the fly or just store cluster aggregates in their own row as well.  If
the latter, depending on your polling methodology, you may want to use
counters for the cluster aggregates.

Also, if you're doing 5 min intervals with each row = 1 day, then your
disk space usage is going to grow pretty quickly due to per-column
overhead.   You didn't say what the values are that you're storing,
but if they're just 64bit integers or something like that, most of
your disk space is actually being used for column overhead not your

I worked around this by creating a 2nd CF, where each row = 1 year
worth of data and each column = 1 days worth of data.  The values are
just a vector of the 5min values from the original CF.  Then I just
have a cron job which reads the previous days data and builds the
vectors in the new CF and then deletes the original row.  By doing
this, my disk space requirements (before replication) went from over
1.1TB/year to 305GB/year.

Aaron Turner         Twitter: @synfinatic - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
    -- Benjamin Franklin
"carpe diem quam minimum credula postero"

View raw message