cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tylerho...@apache.org
Subject [2/3] git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Date Tue, 16 Sep 2014 18:23:06 GMT
Merge branch 'cassandra-2.0' into cassandra-2.1

Conflicts:
	CHANGES.txt
	src/java/org/apache/cassandra/cql3/QueryProcessor.java
	src/java/org/apache/cassandra/metrics/CqlStatementMetrics.java


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/681c380b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/681c380b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/681c380b

Branch: refs/heads/trunk
Commit: 681c380b5a9d0f57b85523a9c23c02e71417c799
Parents: e25d94e 19c6cc1
Author: Tyler Hobbs <tyler@datastax.com>
Authored: Tue Sep 16 13:21:36 2014 -0500
Committer: Tyler Hobbs <tyler@datastax.com>
Committed: Tue Sep 16 13:21:36 2014 -0500

----------------------------------------------------------------------
 CHANGES.txt                                     |  1 +
 .../apache/cassandra/cql3/QueryProcessor.java   | 53 +++++++++++++-------
 .../cassandra/metrics/CqlStatementMetrics.java  |  1 +
 3 files changed, 36 insertions(+), 19 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/cassandra/blob/681c380b/CHANGES.txt
----------------------------------------------------------------------
diff --cc CHANGES.txt
index f89cc6d,cf7112c..0abc66a
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,92 -1,10 +1,93 @@@
 -2.0.11:
 +2.1.1
 + * Fix saving caches when a table is dropped (CASSANDRA-7784)
 + * Add better error checking of new stress profile (CASSANDRA-7716)
 + * Use ThreadLocalRandom and remove FBUtilities.threadLocalRandom (CASSANDRA-7934)
 + * Prevent operator mistakes due to simultaneous bootstrap (CASSANDRA-7069)
 + * cassandra-stress supports whitelist mode for node config (CASSANDRA-7658)
 + * GCInspector more closely tracks GC; cassandra-stress and nodetool report it (CASSANDRA-7916)
 + * nodetool won't output bogus ownership info without a keyspace (CASSANDRA-7173)
 + * Add human readable option to nodetool commands (CASSANDRA-5433)
 + * Don't try to set repairedAt on old sstables (CASSANDRA-7913)
 + * Add metrics for tracking PreparedStatement use (CASSANDRA-7719)
 + * (cqlsh) tab-completion for triggers (CASSANDRA-7824)
 + * (cqlsh) Support for query paging (CASSANDRA-7514)
 + * (cqlsh) Show progress of COPY operations (CASSANDRA-7789)
 + * Add syntax to remove multiple elements from a map (CASSANDRA-6599)
 + * Support non-equals conditions in lightweight transactions (CASSANDRA-6839)
 + * Add IF [NOT] EXISTS to create/drop triggers (CASSANDRA-7606)
 + * (cqlsh) Display the current logged-in user (CASSANDRA-7785)
 + * (cqlsh) Don't ignore CTRL-C during COPY FROM execution (CASSANDRA-7815)
 + * (cqlsh) Order UDTs according to cross-type dependencies in DESCRIBE
 +   output (CASSANDRA-7659)
 + * (cqlsh) Fix handling of CAS statement results (CASSANDRA-7671)
 + * (cqlsh) COPY TO/FROM improvements (CASSANDRA-7405)
 + * Support list index operations with conditions (CASSANDRA-7499)
 + * Add max live/tombstoned cells to nodetool cfstats output (CASSANDRA-7731)
 + * Validate IPv6 wildcard addresses properly (CASSANDRA-7680)
 + * (cqlsh) Error when tracing query (CASSANDRA-7613)
 + * Avoid IOOBE when building SyntaxError message snippet (CASSANDRA-7569)
 + * SSTableExport uses correct validator to create string representation of partition
 +   keys (CASSANDRA-7498)
 + * Avoid NPEs when receiving type changes for an unknown keyspace (CASSANDRA-7689)
 + * Add support for custom 2i validation (CASSANDRA-7575)
 + * Pig support for hadoop CqlInputFormat (CASSANDRA-6454)
 + * Add listen_interface and rpc_interface options (CASSANDRA-7417)
 + * Improve schema merge performance (CASSANDRA-7444)
 + * Adjust MT depth based on # of partition validating (CASSANDRA-5263)
 + * Optimise NativeCell comparisons (CASSANDRA-6755)
 + * Configurable client timeout for cqlsh (CASSANDRA-7516)
 + * Include snippet of CQL query near syntax error in messages (CASSANDRA-7111)
 +Merged from 2.0:
+  * Add metrics for prepared statement usage and eviction (CASSANDRA-7930)
   * Make CQLSSTableWriter sync within partitions (CASSANDRA-7360)
   * Potentially use non-local replicas in CqlConfigHelper (CASSANDRA-7906)
 - * Explicitly disallowing mixing multi-column and single-column
 + * Explicitly disallow mixing multi-column and single-column
     relations on clustering columns (CASSANDRA-7711)
   * Better error message when condition is set on PK column (CASSANDRA-7804)
 + * Don't send schema change responses and events for no-op DDL
 +   statements (CASSANDRA-7600)
 + * (Hadoop) fix cluster initialisation for a split fetching (CASSANDRA-7774)
 + * Throw InvalidRequestException when queries contain relations on entire
 +   collection columns (CASSANDRA-7506)
 + * (cqlsh) enable CTRL-R history search with libedit (CASSANDRA-7577)
 + * (Hadoop) allow ACFRW to limit nodes to local DC (CASSANDRA-7252)
 + * (cqlsh) cqlsh should automatically disable tracing when selecting
 +   from system_traces (CASSANDRA-7641)
 + * (Hadoop) Add CqlOutputFormat (CASSANDRA-6927)
 + * Don't depend on cassandra config for nodetool ring (CASSANDRA-7508)
 + * (cqlsh) Fix failing cqlsh formatting tests (CASSANDRA-7703)
 + * Fix IncompatibleClassChangeError from hadoop2 (CASSANDRA-7229)
 + * Add 'nodetool sethintedhandoffthrottlekb' (CASSANDRA-7635)
 + * (cqlsh) Add tab-completion for CREATE/DROP USER IF [NOT] EXISTS (CASSANDRA-7611)
 + * Catch errors when the JVM pulls the rug out from GCInspector (CASSANDRA-5345)
 + * cqlsh fails when version number parts are not int (CASSANDRA-7524)
 +Merged from 1.2:
 + * Don't index tombstones (CASSANDRA-7828)
 + * Improve PasswordAuthenticator default super user setup (CASSANDRA-7788)
 +
 +
 +2.1.0
 + * (cqlsh) Removed "ALTER TYPE <name> RENAME TO <name>" from tab-completion
 +   (CASSANDRA-7895)
 + * Fixed IllegalStateException in anticompaction (CASSANDRA-7892)
 + * cqlsh: DESCRIBE support for frozen UDTs, tuples (CASSANDRA-7863)
 + * Avoid exposing internal classes over JMX (CASSANDRA-7879)
 + * Add null check for keys when freezing collection (CASSANDRA-7869)
 + * Improve stress workload realism (CASSANDRA-7519)
 +
 +
 +2.1.0-rc7
 + * Add frozen keyword and require UDT to be frozen (CASSANDRA-7857)
 + * Track added sstable size correctly (CASSANDRA-7239)
 + * (cqlsh) Fix case insensitivity (CASSANDRA-7834)
 + * Fix failure to stream ranges when moving (CASSANDRA-7836)
 + * Correctly remove tmplink files (CASSANDRA-7803)
 + * (cqlsh) Fix column name formatting for functions, CAS operations,
 +   and UDT field selections (CASSANDRA-7806)
 + * (cqlsh) Fix COPY FROM handling of null/empty primary key
 +   values (CASSANDRA-7792)
 + * Fix ordering of static cells (CASSANDRA-7763)
 +Merged from 2.0:
   * Forbid re-adding dropped counter columns (CASSANDRA-7831)
   * Fix CFMetaData#isThriftCompatible() for PK-only tables (CASSANDRA-7832)
   * Always reject inequality on the partition key without token()

http://git-wip-us.apache.org/repos/asf/cassandra/blob/681c380b/src/java/org/apache/cassandra/cql3/QueryProcessor.java
----------------------------------------------------------------------
diff --cc src/java/org/apache/cassandra/cql3/QueryProcessor.java
index 99972a2,ee188a3..c983d72
--- a/src/java/org/apache/cassandra/cql3/QueryProcessor.java
+++ b/src/java/org/apache/cassandra/cql3/QueryProcessor.java
@@@ -19,10 -19,9 +19,12 @@@ package org.apache.cassandra.cql3
  
  import java.nio.ByteBuffer;
  import java.util.*;
 +import java.util.concurrent.ConcurrentMap;
 +import java.util.concurrent.ConcurrentHashMap;
+ import java.util.concurrent.TimeUnit;
+ import java.util.concurrent.atomic.AtomicLong;
  
 +import com.google.common.annotations.VisibleForTesting;
  import com.google.common.primitives.Ints;
  
  import com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap;
@@@ -36,13 -34,11 +38,15 @@@ import org.slf4j.LoggerFactory
  
  import org.apache.cassandra.cql3.statements.*;
  import org.apache.cassandra.db.*;
 +import org.apache.cassandra.db.composites.*;
 +import org.apache.cassandra.db.marshal.AbstractType;
  import org.apache.cassandra.exceptions.*;
+ import org.apache.cassandra.metrics.CqlStatementMetrics;
  import org.apache.cassandra.service.ClientState;
  import org.apache.cassandra.service.QueryState;
 +import org.apache.cassandra.service.pager.QueryPager;
 +import org.apache.cassandra.service.pager.QueryPagers;
+ import org.apache.cassandra.service.StorageService;
  import org.apache.cassandra.thrift.ThriftClientState;
  import org.apache.cassandra.tracing.Tracing;
  import org.apache.cassandra.transport.messages.ResultMessage;
@@@ -78,72 -75,67 +82,87 @@@ public class QueryProcessor implements 
          }
      };
  
 -    private static final ConcurrentLinkedHashMap<MD5Digest, CQLStatement> preparedStatements;
 +    private static final ConcurrentLinkedHashMap<MD5Digest, ParsedStatement.Prepared>
preparedStatements;
      private static final ConcurrentLinkedHashMap<Integer, CQLStatement> thriftPreparedStatements;
  
 +    // A map for prepared statements used internally (which we don't want to mix with user
statement, in particular we don't
 +    // bother with expiration on those.
 +    private static final ConcurrentMap<String, ParsedStatement.Prepared> internalStatements
= new ConcurrentHashMap<>();
 +
 +    @VisibleForTesting
      public static final CqlStatementMetrics metrics = new CqlStatementMetrics();
+     private static AtomicLong evictionCount = new AtomicLong(0);
  
      static
      {
 -        if (MemoryMeter.isInitialized())
 -        {
 -            preparedStatements = new ConcurrentLinkedHashMap.Builder<MD5Digest, CQLStatement>()
 -                                 .maximumWeightedCapacity(MAX_CACHE_PREPARED_MEMORY)
 -                                 .weigher(cqlMemoryUsageWeigher)
 -                                 .listener(new EvictionListener<MD5Digest, CQLStatement>()
 -                                  {
 -                                      @Override
 -                                      public void onEviction(MD5Digest md5Digest, CQLStatement
prepared)
 -                                      {
 -                                          metrics.activePreparedStatements.dec();
 -                                          metrics.evictedPreparedStatements.inc();
 -                                          evictionCount.incrementAndGet();
 -                                      }
 -                                  }).build();
 -            thriftPreparedStatements = new ConcurrentLinkedHashMap.Builder<Integer, CQLStatement>()
 -                                       .maximumWeightedCapacity(MAX_CACHE_PREPARED_MEMORY)
 -                                       .weigher(thriftMemoryUsageWeigher)
 -                                       .listener(new EvictionListener<Integer, CQLStatement>()
 -                                        {
 -                                            @Override
 -                                            public void onEviction(Integer i, CQLStatement
prepared)
 -                                            {
 -                                                metrics.activePreparedStatements.dec();
 -                                                metrics.evictedPreparedStatements.inc();
 -                                                evictionCount.incrementAndGet();
 -                                            }
 -                                        }).build();
 -        }
 -        else
 -        {
 -            logger.error("Unable to initialize MemoryMeter (jamm not specified as javaagent).
 This means "
 -                         + "Cassandra will be unable to measure object sizes accurately
and may consequently OOM.");
 -            preparedStatements = new ConcurrentLinkedHashMap.Builder<MD5Digest, CQLStatement>()
 -                                 .maximumWeightedCapacity(MAX_CACHE_PREPARED_COUNT)
 -                                 .build();
 -            thriftPreparedStatements = new ConcurrentLinkedHashMap.Builder<Integer, CQLStatement>()
 -                                       .maximumWeightedCapacity(MAX_CACHE_PREPARED_COUNT)
 -                                       .build();
 -        }
 +        preparedStatements = new ConcurrentLinkedHashMap.Builder<MD5Digest, ParsedStatement.Prepared>()
 +                             .maximumWeightedCapacity(MAX_CACHE_PREPARED_MEMORY)
 +                             .weigher(cqlMemoryUsageWeigher)
 +                             .listener(new EvictionListener<MD5Digest, ParsedStatement.Prepared>()
 +                             {
 +                                 @Override
 +                                 public void onEviction(MD5Digest md5Digest, ParsedStatement.Prepared
prepared)
 +                                 {
 +                                     metrics.activePreparedStatements.dec();
++                                     metrics.evictedPreparedStatements.inc();
++                                     evictionCount.incrementAndGet();
 +                                 }
 +                             }).build();
 +
 +        thriftPreparedStatements = new ConcurrentLinkedHashMap.Builder<Integer, CQLStatement>()
 +                                   .maximumWeightedCapacity(MAX_CACHE_PREPARED_MEMORY)
 +                                   .weigher(thriftMemoryUsageWeigher)
 +                                   .listener(new EvictionListener<Integer, CQLStatement>()
 +                                   {
 +                                       @Override
 +                                       public void onEviction(Integer integer, CQLStatement
cqlStatement)
 +                                       {
 +                                           metrics.activePreparedStatements.dec();
++                                           metrics.evictedPreparedStatements.inc();
++                                           evictionCount.incrementAndGet();
 +                                       }
 +                                   })
 +                                   .build();
  
+         StorageService.scheduledTasks.scheduleAtFixedRate(new Runnable() {
+             @Override
+             public void run() {
+                 long count = evictionCount.getAndSet(0);
+                 if (count > 0)
+                 {
+                     logger.info("{} prepared statements discarded in the last minute because
cache limit reached (cache limit = {} bytes)", count, MAX_CACHE_PREPARED_MEMORY);
+                 }
+             }
+         }, 1, 1, TimeUnit.MINUTES);
      }
  
 +    // Work aound initialization dependency
 +    private static enum InternalStateInstance
 +    {
 +        INSTANCE;
 +
 +        private final QueryState queryState;
 +
 +        InternalStateInstance()
 +        {
 +            ClientState state = ClientState.forInternalCalls();
 +            try
 +            {
 +                state.setKeyspace(Keyspace.SYSTEM_KS);
 +            }
 +            catch (InvalidRequestException e)
 +            {
 +                throw new RuntimeException();
 +            }
 +            this.queryState = new QueryState(state);
 +        }
 +    }
 +
 +    private static QueryState internalQueryState()
 +    {
 +        return InternalStateInstance.INSTANCE.queryState;
 +    }
 +
      private QueryProcessor()
      {
      }
@@@ -394,28 -306,26 +413,24 @@@
              throw new InvalidRequestException(String.format("Prepared statement of size
%d bytes is larger than allowed maximum of %d bytes.",
                                                              statementSize,
                                                              MAX_CACHE_PREPARED_MEMORY));
-         try
 -
+         if (forThrift)
          {
-             if (forThrift)
-             {
-                 int statementId = toHash.hashCode();
-                 thriftPreparedStatements.put(statementId, prepared.statement);
-                 logger.trace("Stored prepared statement #{} with {} bind markers",
-                         statementId,
-                         prepared.statement.getBoundTerms());
-                 return ResultMessage.Prepared.forThrift(statementId, prepared.boundNames);
-             } else
-             {
-                 MD5Digest statementId = MD5Digest.compute(toHash);
-                 preparedStatements.put(statementId, prepared);
-                 logger.trace("Stored prepared statement #{} with {} bind markers",
-                         statementId,
-                         prepared.statement.getBoundTerms());
-                 return new ResultMessage.Prepared(statementId, prepared);
-             }
-         } finally
+             int statementId = toHash.hashCode();
+             thriftPreparedStatements.put(statementId, prepared.statement);
+             metrics.activePreparedStatements.inc();
 -            logger.trace(String.format("Stored prepared statement #%d with %d bind markers",
 -                                       statementId,
 -                                       prepared.statement.getBoundTerms()));
++            logger.trace("Stored prepared statement #{} with {} bind markers",
++                    statementId,
++                    prepared.statement.getBoundTerms());
+             return ResultMessage.Prepared.forThrift(statementId, prepared.boundNames);
 -        }
 -        else
++        } else
          {
+             MD5Digest statementId = MD5Digest.compute(toHash);
 -            preparedStatements.put(statementId, prepared.statement);
++            preparedStatements.put(statementId, prepared);
              metrics.activePreparedStatements.inc();
 -            logger.trace(String.format("Stored prepared statement %s with %d bind markers",
 -                                       statementId,
 -                                       prepared.statement.getBoundTerms()));
++            logger.trace("Stored prepared statement #{} with {} bind markers",
++                    statementId,
++                    prepared.statement.getBoundTerms());
+             return new ResultMessage.Prepared(statementId, prepared);
          }
      }
  


Mime
View raw message