Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of jbellis@gmail.com designates
 74.125.82.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=jgd/Thm+BVdOGLIqEpCTVbUDyVTRS/59G70an+5IuG6P90l2ZbD8y7s3uU9Cze8yc/
         X5UxZajvGABHjI78c+L3lh17Z7X560uBftn6jVOyvAzurYnX6hjwdiL7nzVNPJz0575t
         7/DaA5g8Td1yYLJe0hv0SzxT7MHf8quNe48NM=
MIME-Version: 1.0
In-Reply-To: <q2t72c7a7d41005042053hf2ef2158ze5466b124910129e@mail.gmail.com>
References: <q2t72c7a7d41005042053hf2ef2158ze5466b124910129e@mail.gmail.com>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Wed, 5 May 2010 09:23:20 -0500
Message-ID: <k2te06563881005050723zc71595ear8545594857f07418@mail.gmail.com>
Subject: Re: Updating (as opposed to just setting) Cassandra data via Hadoop
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I'm a little confused.  CombineFileInputFormat is designed to combine
multiple small input splits into one larger one.  It's not for merging
data (that needs to be part of the reduce phase).  Maybe I'm
misunderstanding what you're saying.

On Tue, May 4, 2010 at 10:53 PM, Mark Schnitzius
<mark.schnitzius@cxense.com> wrote:
> I have a situation where I need to accumulate values in Cassandra on an
> ongoing basis. =A0Atomic increments are still in the works apparently
> (see=A0https://issues.apache.org/jira/browse/CASSANDRA-721) so for the ti=
me
> being I'll be using Hadoop, and attempting to feed in both the existing
> values and the new values to a M/R process where they can be combined
> together and written back out to Cassandra.
> The approach I'm taking is to use Hadoop's CombineFileInputFormat to blen=
d
> the existing data (using Cassandra's ColumnFamilyInputFormat) with the ne=
wly
> incoming data (using something like Hadoop's SequenceFileInputFormat).
> I was just wondering, has anyone here tried this, and were there issues?
> =A0I'm worried because the=A0CombineFileInputFormat has restrictions arou=
nd
> splits being from different pools so I don't know how this will play out
> with data from both Cassandra and HDFS. =A0The other option, I suppose, i=
s to
> use a separate M/R process to replicate the data onto HDFS first, but I'd
> rather avoid the extra step and duplication of storage.
> Also, if you've tackled a similar situation in the past using a different
> approach, I'd be keen to hear about it...
>
> Thanks
> Mark


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com