cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dinesh.simkh...@gridcore.se
Subject Re: Cassandra data model help
Date Fri, 10 Aug 2012 08:34:29 GMT
Thanks Aaron for your reply,
creating vector for raw data is good work around for decreasing disk space, but I am not still
clear tracking time for nodes, say if we want a query like give me the list of nodes for a
cluster between this period of time then how do we get that information? do we scan through
each node row as we will have row for each node? 

thanks

-----Aaron Turner <synfinatic@gmail.com> wrote: -----
To: user@cassandra.apache.org
From: Aaron Turner <synfinatic@gmail.com>
Date: 08/09/2012 07:38PM
Subject: Re: Cassandra data model help

On Thu, Aug 9, 2012 at 5:52 AM,  <dinesh.simkhada@gridcore.se> wrote:
> Hi,
> I am trying to create a Cassandra schema for cluster monitoring system, where one cluster
can have multiple nodes and I am monitoring multiple matrices from a node. My raw data schema
looks like and taking values in every 5 min interval
>
> matrix_name + daily time stamp as row key, composite column name of node name and time
stamp and matrix value as column value
>
> the problem I am facing is a node can go back and forth between the clusters(system can
have more than one clusters) so if i need monthly statistics plotting of a cluster I have
to consider the nodes that are leaving and joining during this period of time, some node might
be part of the cluster for just 15 days and some could join the cluster last 10 day of month,
so to plot data for a particular cluster for a time interval I need to know the nodes which
were part of that cluster for that period of time, what could be the best schema for this
solution ? I have tried few ideas so far no luck, any suggestions ?

Store each node stat in it's own row.  Then decide if you want to
track when a node joins/leaves a cluster so you can build the aggs on
the fly or just store cluster aggregates in their own row as well.  If
the latter, depending on your polling methodology, you may want to use
counters for the cluster aggregates.

Also, if you're doing 5 min intervals with each row = 1 day, then your
disk space usage is going to grow pretty quickly due to per-column
overhead.   You didn't say what the values are that you're storing,
but if they're just 64bit integers or something like that, most of
your disk space is actually being used for column overhead not your
data.

I worked around this by creating a 2nd CF, where each row = 1 year
worth of data and each column = 1 days worth of data.  The values are
just a vector of the 5min values from the original CF.  Then I just
have a cron job which reads the previous days data and builds the
vectors in the new CF and then deletes the original row.  By doing
this, my disk space requirements (before replication) went from over
1.1TB/year to 305GB/year.


-- 
Aaron Turner
http://synfin.net/         Twitter: @synfinatic
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
    -- Benjamin Franklin
"carpe diem quam minimum credula postero"


Mime
View raw message