incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Hanna <jeremy.hanna1...@gmail.com>
Subject Re: Massive writes when only reading from Cassandra
Date Sat, 10 Sep 2011 22:14:57 GMT
Thanks for the insights.  I may first try disabling hinted handoff for one run of our data
pipeline and see if it exhibits the same behavior.  Will post back if I see anything enlightening
there.

On Sep 10, 2011, at 5:04 PM, Chris Goffinet wrote:

> You could tail the commit log with `strings` to see what keys are being inserted.
> 
> On Sat, Sep 10, 2011 at 2:24 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
> Two possibilities:
> 
> 1) Hinted handoff (this will show up in the logs on the sending
> machine, on the receiving one it will just look like any other write)
> 
> 2) You have something doing writes that you're not aware of, I guess
> you could track that down using wireshark to see where the write
> messages are coming from
> 
> On Sat, Sep 10, 2011 at 3:56 PM, Jeremy Hanna
> <jeremy.hanna1234@gmail.com> wrote:
> > Oh and we're running 0.8.4 and the RF is 3.
> >
> > On Sep 10, 2011, at 3:49 PM, Jeremy Hanna wrote:
> >
> >> In addition, the mutation stage and the read stage are backed up like:
> >>
> >> Pool Name                    Active   Pending   Blocked
> >> ReadStage                        32       773         0
> >> RequestResponseStage              0         0         0
> >> ReadRepairStage                   0         0         0
> >> MutationStage                   158    525918         0
> >> ReplicateOnWriteStage             0         0         0
> >> GossipStage                       0         0         0
> >> AntiEntropyStage                  0         0         0
> >> MigrationStage                    0         0         0
> >> StreamStage                       0         0         0
> >> MemtablePostFlusher               1         5         0
> >> FILEUTILS-DELETE-POOL             0         0         0
> >> FlushWriter                       2         5         0
> >> MiscStage                         0         0         0
> >> FlushSorter                       0         0         0
> >> InternalResponseStage             0         0         0
> >> HintedHandoff                     0         0         0
> >> CompactionManager               n/a        29
> >> MessagingService                n/a      0,34
> >>
> >> On Sep 10, 2011, at 3:38 PM, Jeremy Hanna wrote:
> >>
> >>> We are experiencing massive writes to column families when only doing reads
from Cassandra.  A set of 5 hadoop jobs are reading from Cassandra and then writing out to
hdfs.  That is the only thing operating on the cluster.  We are reading at CL.QUORUM with
hadoop and have written with CL.QUORUM.  Read repair chance is set to 0.0 on all column families.
 However, in the logs, I'm seeing flush after flush of memtables and compactions taking place.
 Is there something else that would be writing based on the above description?
> >>>
> >>> Jeremy
> >>
> >
> >
> 
> 
> 
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
> 


Mime
View raw message