Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of janne.jalkanen@ecyrd.com
 designates 87.108.86.67 as permitted sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\))
Subject: Re: Billions of counters
From: Janne Jalkanen <janne.jalkanen@ecyrd.com>
In-Reply-To: 
 <CAN94HvOmwJ0+arokUtnyTrsoxzN=C726GPaBhBxY+UtgGk7=Sw@mail.gmail.com>
Date: Thu, 13 Jun 2013 21:51:30 +0300
Content-Transfer-Encoding: quoted-printable
Message-Id: <392FACAC-DA63-4CBE-9B30-92F873817685@ecyrd.com>
References: 
 <CAN94HvOmwJ0+arokUtnyTrsoxzN=C726GPaBhBxY+UtgGk7=Sw@mail.gmail.com>
To: user@cassandra.apache.org


Hi!

We have a similar situation of millions of events on millions of items - =
turns out that this isn't really a problem, because there tends to be a =
very strong power -distribution: very few of the items get a lot of =
hits, some get some, and the majority gets no hits (though most of them =
do get hits every now and then).  So it's basically a sparse =
multidimensional array, and turns out that Cassandra is pretty good at =
storing those.  We just treat a missing counter column as zero, and add =
a counter only when necessary.  To avoid I/O, we also do some =
statistical sampling for certain counters where we don't need an exact =
figure.

YMMV, of course, but I'd look at the likelihood of all the products =
being purchased from the same location during one week at least once and =
start the modeling from there. :)

/Janne

On 13 Jun 2013, at 21:19, Darren Smythe <darren1482@gmail.com> wrote:

> We want to precalculate counts for some common metrics for usage. We =
have events, locations, products, etc. The problem is we have millions =
events/day, thousands of locations and millions of products.
>=20
> Were trying to precalculate counts for some common queries like 'how =
many times was product X purchased in location Y last week'.
>=20
> It seems like we'll end up with trillions of counters for even these =
basic permutations. Is this a cause for concern?
>=20
> TIA
>=20
> -- Darren