Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jeremy.hanna1234@gmail.com
 designates 209.85.219.44 as permitted sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1486\))
Subject: Re: Differences in row iteration behavior
From: Jeremy Hanna <jeremy.hanna1234@gmail.com>
In-Reply-To: <5053F0DF.8050500@conga.com>
Date: Fri, 14 Sep 2012 22:21:29 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <B62545D9-267A-431A-AB84-53367A606ACA@gmail.com>
References: <5053F0DF.8050500@conga.com>
To: user@cassandra.apache.org

Are there any deletions in your data?  The Hadoop support doesn't filter =
out tombstones, though you may not be filtering them out in your code =
either.  I've used the hadoop support for doing a lot of data validation =
in the past and as long as you're sure that the code is sound, I'm =
pretty confident in it.

On Sep 14, 2012, at 10:07 PM, Todd Fast <todd@conga.com> wrote:

> Hi--
>=20
> We are iterating rows in a column family two different ways and are =
seeing radically different row counts. We are using 1.0.8 and =
RandomPartitioner on a 3-node cluster.
>=20
> In the first case, we have a trivial Hadoop job that counts 29M rows =
using the standard MR pattern for counting (mapper outputs a single key =
with a value of 1, reducer adds up all the values).
>=20
> In the second case, we have a simple Quartz batch job which counts =
only 10M rows. We are iterating using chained calls to get_row_slices, =
as described on the wiki: =
http://wiki.apache.org/cassandra/FAQ#iter_world We've also implemented =
the batch job using Pelops, with and without chaining. In all cases, the =
job counts just 10M rows, and it is not encountering any errors.
>=20
> We are confident that we are doing everything right in both cases (no =
bugs), yet the results are baffling. Tests in smaller, single-node =
environments results in consistent counts between the two methods, but =
we don't have the same amount of data nor the same topology.
>=20
> Is the right answer 29M or 10M? Any clues to what we're seeing?
>=20
> Todd