incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Read operations resulting in a write?
Date Tue, 18 Dec 2012 05:10:48 GMT
Is there a way to turn this on and off through configuration? I am not
necessarily sure I would want this feature. Also it is confusing if these
writes show up in JMX and look like user generated write operations.


On Mon, Dec 17, 2012 at 10:01 AM, Mike <mtheroux2@yahoo.com> wrote:

>  Thank you Aaron, this was very helpful.
>
> Could it be an issue that this optimization does not really take effect
> until the memtable with the hoisted data is flushed?  In my simple example
> below, the same row is updated and multiple selects of the same row will
> result in multiple writes to the memtable.  It seems it maybe possible
> (although unlikely) where, if you go from a write-mostly to a read-mostly
> scenario, you could get into a state where you are stuck rewriting to the
> same memtable, and the memtable is not flushed because it absorbs the
> over-writes.  I can foresee this especially if you are reading the same
> rows repeatedly.
>
> I also noticed from the codepaths that if Row caching is enabled, this
> optimization will not occur.  We made some changes this weekend to make
> this column family more suitable to row-caching and enabled row-caching
> with a small cache.  Our initial results is that it seems to have corrected
> the write counts, and has increased performance quite a bit.  However, are
> there any hidden gotcha's there because this optimization is not
> occurring?  https://issues.apache.org/jira/browse/CASSANDRA-2503 mentions
> a "compaction is behind" problem.  Any history on that?  I couldn't find
> too much information on it.
>
> Thanks,
> -Mike
>
> On 12/16/2012 8:41 PM, aaron morton wrote:
>
>
>   1) Am I reading things correctly?
>
> Yes.
> If you do a read/slice by name and more than min compaction level nodes
> where read the data is re-written so that the next read uses fewer SSTables.
>
>  2) What is really happening here?  Essentially minor compactions can
> occur between 4 and 32 memtable flushes.  Looking through the code, this
> seems to only effect a couple types of select statements (when selecting a
> specific column on a specific key being one of them). During the time
> between these two values, every "select" statement will perform a write.
>
> Yup, only for readying a row where the column names are specified.
> Remember minor compaction when using SizedTiered Compaction (the default)
> works on buckets of the same size.
>
>  Imagine a row that had been around for a while and had fragments in more
> than Min Compaction Threshold sstables. Say it is 3 SSTables in the 2nd
> tier and 2 sstables in the 1st. So it takes (potentially) 5 SSTable reads.
> If this row is read it will get hoisted back up.
>
>  But the row has is in only 1 SSTable in the 2nd tier and 2 in the 1st
> tier it will not hoisted.
>
>  There are a few short circuits in the SliceByName read path. One of them
> is to end the search when we know that no other SSTables contain columns
> that should be considered. So if the 4 columns you read frequently are
> hoisted into the 1st bucket your reads will get handled by that one bucket.
>
>  It's not every select. Just those that touched more the min compaction
> sstables.
>
>
>  3) Is this desired behavior?  Is there something else I should be
> looking at that could be causing this behavior?
>
> Yes.
> https://issues.apache.org/jira/browse/CASSANDRA-2503
>
>  Cheers
>
>
>    -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
>  @aaronmorton
> http://www.thelastpickle.com
>
>  On 15/12/2012, at 12:58 PM, Michael Theroux <mtheroux2@yahoo.com> wrote:
>
>  Hello,
>
>  We have an unusual situation that I believe I've reproduced, at least
> temporarily, in a test environment.  I also think I see where this issue is
> occurring in the code.
>
>  We have a specific column family that is under heavy read and write load
> on a nightly basis.   For the purposes of this description, I'll refer to
> this column family as "Bob".  During this nightly processing, sometimes Bob
> is under very write load, other times it is very heavy read load.
>
>  The application is such that when something is written to Bob, a write
> is made to one of two other tables.  We've witnessed a situation where the
> write count on Bob far outstrips the write count on either of the other
> tables, by a factor of 3->10.  This is based on the WriteCount available on
> the column family JMX MBean.  We have not been able to find where in our
> code this is happening, and we have gone as far as tracing our CQL calls to
> determine that the relationship between Bob and the other tables are what
> we expect.
>
>  I brought up a test node to experiment, and see a situation where, when
> a "select" statement is executed, a write will occur.
>
>  In my test, I perform the following (switching between nodetool and
> cqlsh):
>
>  update bob set 'about'='coworker' where key='<hex key>';
> nodetool flush
>  update bob set 'about'='coworker' where key='<hex key>';
> nodetool flush
>  update bob set 'about'='coworker' where key='<hex key>';
> nodetool flush
>  update bob set 'about'='coworker' where key='<hex key>';
> nodetool flush
>  update bob set 'about'='coworker' where key='<hex key>';
> nodetool flush
>
>  Then, for a period of time (before a minor compaction occurs), a select
> statement that selects specific columns will cause writes to occur in the
> write count of the column family:
>
>  select about,changed,data from bob where key='<hex key>';
>
>  This situation will continue until a minor compaction is completed.
>
>  I went into the code and added some traces to CollationController.java:
>
>    private ColumnFamily collectTimeOrderedData()
>     {
>         logger.debug("collectTimeOrderedData");
>
>       ... <snip> ...
>
>
> ---> HERE   logger.debug( "tables iterated: " + sstablesIterated +  " Min compact:
" + cfs.getMinimumCompactionThreshold() );
>
>             // "hoist up" the requested data into a more recent sstable
>             if (sstablesIterated > cfs.getMinimumCompactionThreshold()
>                 && !cfs.isCompactionDisabled()
>                 && cfs.getCompactionStrategy() instanceof SizeTieredCompactionStrategy)
>             {
>                 RowMutation rm = new RowMutation(cfs.table.name, new Row(filter.key,
returnCF.cloneMe()));
>                 try
>                 {
>
> ---> HERE               logger.debug( "Apply hoisted up row mutation" );	
>
>                     // skipping commitlog and index updates is fine since we're just
de-fragmenting existing data
>                     Table.open(rm.getTable()).apply(rm, false, false);
>                 }
>                 catch (IOException e)
>                 {
>                     // log and allow the result to be returned
>                     logger.error("Error re-writing read results", e);
>                 }
>             }
>
>  ... <snip> ...
>
>  Performing the steps above, I see the following traces (in the test
> environment I decreased the minimum compaction threshold to make this
> easier to reproduce). After I do a couple of update/flush, I see this in
> the log:
>
>  DEBUG [FlushWriter:7] 2012-12-14 22:54:40,106 CompactionManager.java
> (line 117) Scheduling a background task check for bob with
> SizeTieredCompactionStrategy
>
>  Then, until compaction occurs, I see (when performing a select):
>
>  DEBUG [ScheduledTasks:1] 2012-12-14 22:55:15,998 LoadBroadcaster.java
> (line 86) Disseminating load info ...
> DEBUG [Thrift:12] 2012-12-14 22:55:16,990 CassandraServer.java (line 1227)
> execute_cql_query
> DEBUG [Thrift:12] 2012-12-14 22:55:16,991 QueryProcessor.java (line 445)
> CQL statement type: SELECT
> DEBUG [Thrift:12] 2012-12-14 22:55:16,991 StorageProxy.java (line 653)
> Command/ConsistencyLevel is SliceByNamesReadCommand(table='open',
> key=804229d1933669d0a25d2a38c8b26ded10069573003e6dbb1ce21b5f402a5342,
> columnParent='QueryPath(columnFamilyName='bob', superColumnName='null',
> columnName='null')', columns=[about,changed,data,])/ONE
> DEBUG [Thrift:12] 2012-12-14 22:55:16,992 ReadCallback.java (line 79)
> Blockfor is 1; setting up requests to /10.0.4.20
> DEBUG [Thrift:12] 2012-12-14 22:55:16,992 StorageProxy.java (line 669)
> reading data locally
> DEBUG [ReadStage:61] 2012-12-14 22:55:16,992 StorageProxy.java (line 813)
> LocalReadRunnable reading SliceByNamesReadCommand(table='open',
> key=804229d1933669d0a25d2a38c8b26ded10069573003e6dbb1ce21b5f402a5342,
> columnParent='QueryPath(columnFamilyName='bob', superColumnName='null',
> columnName='null')', columns=[about,changed,data,])
> DEBUG [ReadStage:61] 2012-12-14 22:55:16,992 CollationController.java
> (line 68) In get top level columns: class
> org.apache.cassandra.db.filter.NamesQueryFilter type: Standard valid: class
> org.apache.cassandra.db.marshal.BytesType
> DEBUG [ReadStage:61] 2012-12-14 22:55:16,992 CollationController.java
> (line 84) collectTimeOrderedData
> ---> DEBUG [ReadStage:61] 2012-12-14 22:55:17,192 CollationController.java
> (line 188) tables iterated: 4 Min compact: 2
>  ----> DEBUG [ReadStage:61] 2012-12-14 22:55:17,192
> CollationController.java (line 198) Apply hoisted up row mutation
> DEBUG [ReadStage:61] 2012-12-14 22:55:17,193 Table.java (line 395)
> applying mutation of row
> 804229d1933669d0a25d2a38c8b26ded10069573003e6dbb1ce21b5f402a5342
>
>  The above traces will occur every time I repeat the above select
> statement.
>
>  Minor compaction doesn't start until a few minutes after the request was
> submitted above (note, this is an unloaded test node):
>
>  DEBUG [CompactionExecutor:11] 2012-12-14 22:57:03,278 IntervalNode.java
> (line 45) Creating IntervalNode from
> [Interval(DecoratedKey(Token(bytes[804229d1933669d0a25d2a38c8b26ded10069573003e6dbb1ce...
>
>  Once minor compaction occurs, the behavior around write count being
> incremented stops, until more than the minimum compaction threshold
> memtables are flush to disk.
>
>  So, my questions are:
>
>  1) Am I reading things correctly?
>
>  2) What is really happening here?  Essentially minor compactions can
> occur between 4 and 32 memtable flushes.  Looking through the code, this
> seems to only effect a couple types of select statements (when selecting a
> specific column on a specific key being one of them). During the time
> between these two values, every "select" statement will perform a write.
>
>  3) Is this desired behavior?  Is there something else I should be
> looking at that could be causing this behavior?
>
>  We are running Cassandra 1.1.2, with SizeTieredCompactionStrategy.
> Any help is appreciated,
> Thanks,
> -Mike
>
>
>
>
>
>
>

Mime
View raw message