incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vick Khera <vi...@khera.org>
Subject Re: Re: Modelling assets and user permissions
Date Tue, 20 Apr 2010 19:53:22 GMT
On Tue, Apr 20, 2010 at 1:37 PM, tsuraan <tsuraan@gmail.com> wrote:
> The assets are binary files on a document tracking system.  Our
> current platform is postgres-backed; the entire system we've written
> is fairly easily distributed across multiple computers, but postgres
> isn't.  There are reliable databases that do scale out, but they tend
> to be a little on the pricey side...  Our current system works well in
> the tens to hundreds of millions of documents with hundreds of users,
> but we're hitting the billions of documents with thousands of users,
> so cassandra's scaling properties are pretty appealing there.

It seems to me you might get by with putting the actual assets into
cassandra (possibly breaking them up into chunks depending on how big
they are) and storing the pointers to them in Postgres along with all
the other metadata.  If it were me, I'd split each file into a fixed
chunksize and store it using its SHA1 checksum, and keep an ordered
list of chunks that make up a file, then never delete a chunk.  Given
billions of documents you just may end up with some savings due to
file chunks that are identical.

You could partition the postgres tables and replicate the data to a
handful of read-only nodes that could handle quite a bit of the work.
I suppose it depends on your write-frequency how that might pan out as
a scalability option.

Mime
View raw message