incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yang <teddyyyy...@gmail.com>
Subject Re: MR-hook for sstable compaction?
Date Sun, 19 Jun 2011 19:21:58 GMT
Thanks Jonathan.

yes I did notice the RF issue, and thought , for example, to get a total
salary, you'd need to divide it by RF, something like that.

I'll take a look at 1608,

Yang

On Sun, Jun 19, 2011 at 12:12 AM, Jonathan Ellis <jbellis@gmail.com> wrote:

> I'm skeptical that this is the right place to do M/R jobs (multiple
> replicas mean you'll do the work multiple times, if you have the same
> code on all nodes... and different code on the nodes could get messy
> fast.)
>
> But, the work-in-progress patches on CASSANDRA-1608 include a
> compaction pub/sub component.  So you could create a subclass of your
> desired compaction strategy that adds a "notify reduce job" hook and
> try it out that way.
>
> On Sun, Jun 19, 2011 at 1:59 AM, Yang <teddyyyy123@gmail.com> wrote:
> > I realize that the SSTable flush/compaction process is essentially
> > equivalent to the reduce stage of Map-Reduce,
> > since entries of same keys are grouped together.
> > we have felt the need to do MR-style jobs on the data already stored in
> > cassandra, it would be very useful to
> > provide a hook into the compaction process so that the reduce job can be
> > done. for example, jobs as simple as
> > dumping out all the keys in a system, or for a CF with userId being the
> key,
> > and salary as a column, calculate the total
> > salary.
> > this is different from what BRISK does, since BRISK only uses CF as a
> > physical block storage, and does not
> > utilize the data already stored in Cassandra, which has been grouped by
> > keys.
> > it is possible to come up with some sort of framework to scrape sstables
> to
> > carry out the MR jobs, but the compaction
> > hook seems an easier and faster way to get this done, given existing
> systems
> > yang
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>

Mime
View raw message