Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jeremy.hanna1234@gmail.com
 designates 209.85.160.172 as permitted sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Apple Message framework v1244.3)
Subject: Re: Massive writes when only reading from Cassandra
From: Jeremy Hanna <jeremy.hanna1234@gmail.com>
In-Reply-To: 
 <CA+2nF5Z0ukdqaQBKVGtTAFVQ799UdV24b5vSe5wRdjacy4nUcQ@mail.gmail.com>
Date: Sat, 10 Sep 2011 17:14:57 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <05BCECCE-098A-4205-8C0E-3ADCDF1C0EC2@gmail.com>
References: <95F4FC21-8D11-4375-8503-1A60BD1AD311@gmail.com>
 <1BDB35E4-A61B-4F91-9257-D6F5119F29AB@gmail.com>
 <4ECB8F33-97BE-42D5-A435-707F04EE071B@gmail.com>
 <CALdd-zisafxqwgeS_r0A2_gshgsUtOy3zd-BQMN6jPSE1BtJJQ@mail.gmail.com>
 <CA+2nF5Z0ukdqaQBKVGtTAFVQ799UdV24b5vSe5wRdjacy4nUcQ@mail.gmail.com>
To: user@cassandra.apache.org

Thanks for the insights.  I may first try disabling hinted handoff for =
one run of our data pipeline and see if it exhibits the same behavior.  =
Will post back if I see anything enlightening there.

On Sep 10, 2011, at 5:04 PM, Chris Goffinet wrote:

> You could tail the commit log with `strings` to see what keys are =
being inserted.
>=20
> On Sat, Sep 10, 2011 at 2:24 PM, Jonathan Ellis <jbellis@gmail.com> =
wrote:
> Two possibilities:
>=20
> 1) Hinted handoff (this will show up in the logs on the sending
> machine, on the receiving one it will just look like any other write)
>=20
> 2) You have something doing writes that you're not aware of, I guess
> you could track that down using wireshark to see where the write
> messages are coming from
>=20
> On Sat, Sep 10, 2011 at 3:56 PM, Jeremy Hanna
> <jeremy.hanna1234@gmail.com> wrote:
> > Oh and we're running 0.8.4 and the RF is 3.
> >
> > On Sep 10, 2011, at 3:49 PM, Jeremy Hanna wrote:
> >
> >> In addition, the mutation stage and the read stage are backed up =
like:
> >>
> >> Pool Name                    Active   Pending   Blocked
> >> ReadStage                        32       773         0
> >> RequestResponseStage              0         0         0
> >> ReadRepairStage                   0         0         0
> >> MutationStage                   158    525918         0
> >> ReplicateOnWriteStage             0         0         0
> >> GossipStage                       0         0         0
> >> AntiEntropyStage                  0         0         0
> >> MigrationStage                    0         0         0
> >> StreamStage                       0         0         0
> >> MemtablePostFlusher               1         5         0
> >> FILEUTILS-DELETE-POOL             0         0         0
> >> FlushWriter                       2         5         0
> >> MiscStage                         0         0         0
> >> FlushSorter                       0         0         0
> >> InternalResponseStage             0         0         0
> >> HintedHandoff                     0         0         0
> >> CompactionManager               n/a        29
> >> MessagingService                n/a      0,34
> >>
> >> On Sep 10, 2011, at 3:38 PM, Jeremy Hanna wrote:
> >>
> >>> We are experiencing massive writes to column families when only =
doing reads from Cassandra.  A set of 5 hadoop jobs are reading from =
Cassandra and then writing out to hdfs.  That is the only thing =
operating on the cluster.  We are reading at CL.QUORUM with hadoop and =
have written with CL.QUORUM.  Read repair chance is set to 0.0 on all =
column families.  However, in the logs, I'm seeing flush after flush of =
memtables and compactions taking place.  Is there something else that =
would be writing based on the above description?
> >>>
> >>> Jeremy
> >>
> >
> >
>=20
>=20
>=20
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>=20