Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9631DE503 for ; Tue, 8 Jan 2013 20:42:54 +0000 (UTC) Received: (qmail 19587 invoked by uid 500); 8 Jan 2013 20:42:52 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 19561 invoked by uid 500); 8 Jan 2013 20:42:52 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 19536 invoked by uid 99); 8 Jan 2013 20:42:52 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jan 2013 20:42:52 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a56.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jan 2013 20:42:46 +0000 Received: from homiemail-a56.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a56.g.dreamhost.com (Postfix) with ESMTP id DF215FE059 for ; Tue, 8 Jan 2013 12:42:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=7yCctKNysQXZNSCYZIYtnm6I+o c=; b=DUVIgZHlDqZWqQSnV+3r295IR62ZYsR36j1CQTe2X/wMdgfWAbNVoDE+2/ P2Y9HstLpCk7GbrpTvgb8GXtJ5558A4aMc9PTyNaOsQ3frRWKukoHqrD8tZGEmrR ZMW8138AqBs3iWsgKQTW0gIZ7wPyTMVlArSAHQaFUbY9gefuw= Received: from [172.16.1.8] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a56.g.dreamhost.com (Postfix) with ESMTPSA id 558B3FE05B for ; Tue, 8 Jan 2013 12:42:24 -0800 (PST) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_1B31DA71-E133-4AC4-848C-67848E1BA958" Message-Id: <6A5E739F-29B0-48DD-B518-E12245752206@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: inconsistent hadoop/cassandra results Date: Wed, 9 Jan 2013 09:42:22 +1300 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_1B31DA71-E133-4AC4-848C-67848E1BA958 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Assuming their were no further writes, running repair or using CL all = should have fixed it.=20 Can you describe the inconsistency between runs?=20 Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 8/01/2013, at 2:16 AM, Brian Jeltema = wrote: > I need some help understanding unexpected behavior I saw in some = recent experiments with Cassandra 1.1.5 and Hadoop 1.0.3: >=20 > I've written a small map/reduce job that simply counts the number of = columns in each row of a static CF (call it Foo)=20 > and generates a list of every row and column count. A relatively small = fraction of the rows have a large number > of columns; worst case is approximately 36 million. So when I set up = the job, I used wide-row support: >=20 > ConfigHelper.setInputColumnFamily(job.getConfiguration(), "fooKS", = "Foo", WIDE_ROWS); // where WIDE_ROWS =3D=3D true >=20 > When I ran this job using the default CL (1) I noticed that the = results varied from run to run, which I attributed to inconsistent > replicas, since Foo was generated with CL =3D=3D 1 and the RF =3D=3D = 3.=20 >=20 > So I ran repair for that CF on every node. The cassandra log on every = node contains lines similar to: >=20 > INFO [AntiEntropyStage:1] 2013-01-05 20:38:48,605 = AntiEntropyService.java (line 778) [repair = #e4a1d7f0-579d-11e2-0000-d64e0a75e6df] Foo is fully synced >=20 > However, repeated runs were still inconsistent. Then I set CL to ALL, = which I presumed would always result in identical > output, but repeated runs initially continued to be inconsistent. = However, I noticed that the results seemed to > be converging, and after several runs (somewhere between 4 and 6) I = finally was producing identical results on every run. > Then I set CL to QUORUM, and again generated inconsistent results. >=20 > Does this behavior make sense? >=20 > Brian --Apple-Mail=_1B31DA71-E133-4AC4-848C-67848E1BA958 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
http://www.thelastpickle.com

On 8/01/2013, at 2:16 AM, Brian Jeltema <brian.jeltema@digitalenvoy.= net> wrote:

I need some help = understanding unexpected behavior I saw in some recent experiments with = Cassandra 1.1.5 and Hadoop 1.0.3:

I've written a = small map/reduce job that simply counts the number of columns in each = row of a static CF (call it Foo) 
and generates a list of every = row and column count. A relatively small fraction of the rows have a = large number
of columns; worst case is approximately 36 = million. So when I set up the job, I used wide-row = support:

    ConfigHelper.setInputColumnFamily(job.getConfiguration(), = "fooKS", "Foo", WIDE_ROWS); // where WIDE_ROWS =3D=3D = true
When I ran this job using the default CL (1) I noticed that = the results varied from run to run, which I attributed to = inconsistent
replicas, since Foo was generated with CL =3D=3D 1 and the RF = =3D=3D 3. 

So I ran repair for that CF on every node. The cassandra log = on every node contains lines similar to:

  INFO [AntiEntropyStage:1] = 2013-01-05 20:38:48,605 AntiEntropyService.java (line 778) [repair = #e4a1d7f0-579d-11e2-0000-d64e0a75e6df] Foo is fully = synced

However, repeated runs were still inconsistent. Then I set CL = to ALL, which I presumed would always result in = identical
output, but repeated runs initially continued to be = inconsistent. However, I noticed that the results seemed = to
be converging, and after several runs (somewhere between 4 and = 6) I finally was producing identical results on every = run.
Then I set CL to QUORUM, and again generated inconsistent = results.

Does this behavior make sense?