Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B375111926 for ; Tue, 16 Sep 2014 18:23:05 +0000 (UTC) Received: (qmail 46543 invoked by uid 500); 16 Sep 2014 18:23:05 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 46383 invoked by uid 500); 16 Sep 2014 18:23:05 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 46358 invoked by uid 99); 16 Sep 2014 18:23:05 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Sep 2014 18:23:05 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id 1E4B7A167D4; Tue, 16 Sep 2014 18:23:05 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: tylerhobbs@apache.org To: commits@cassandra.apache.org Date: Tue, 16 Sep 2014 18:23:06 -0000 Message-Id: <9575460bcb0a406a8a34586ad02c01ac@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [2/3] git commit: Merge branch 'cassandra-2.0' into cassandra-2.1 Merge branch 'cassandra-2.0' into cassandra-2.1 Conflicts: CHANGES.txt src/java/org/apache/cassandra/cql3/QueryProcessor.java src/java/org/apache/cassandra/metrics/CqlStatementMetrics.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/681c380b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/681c380b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/681c380b Branch: refs/heads/trunk Commit: 681c380b5a9d0f57b85523a9c23c02e71417c799 Parents: e25d94e 19c6cc1 Author: Tyler Hobbs Authored: Tue Sep 16 13:21:36 2014 -0500 Committer: Tyler Hobbs Committed: Tue Sep 16 13:21:36 2014 -0500 ---------------------------------------------------------------------- CHANGES.txt | 1 + .../apache/cassandra/cql3/QueryProcessor.java | 53 +++++++++++++------- .../cassandra/metrics/CqlStatementMetrics.java | 1 + 3 files changed, 36 insertions(+), 19 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/681c380b/CHANGES.txt ---------------------------------------------------------------------- diff --cc CHANGES.txt index f89cc6d,cf7112c..0abc66a --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,92 -1,10 +1,93 @@@ -2.0.11: +2.1.1 + * Fix saving caches when a table is dropped (CASSANDRA-7784) + * Add better error checking of new stress profile (CASSANDRA-7716) + * Use ThreadLocalRandom and remove FBUtilities.threadLocalRandom (CASSANDRA-7934) + * Prevent operator mistakes due to simultaneous bootstrap (CASSANDRA-7069) + * cassandra-stress supports whitelist mode for node config (CASSANDRA-7658) + * GCInspector more closely tracks GC; cassandra-stress and nodetool report it (CASSANDRA-7916) + * nodetool won't output bogus ownership info without a keyspace (CASSANDRA-7173) + * Add human readable option to nodetool commands (CASSANDRA-5433) + * Don't try to set repairedAt on old sstables (CASSANDRA-7913) + * Add metrics for tracking PreparedStatement use (CASSANDRA-7719) + * (cqlsh) tab-completion for triggers (CASSANDRA-7824) + * (cqlsh) Support for query paging (CASSANDRA-7514) + * (cqlsh) Show progress of COPY operations (CASSANDRA-7789) + * Add syntax to remove multiple elements from a map (CASSANDRA-6599) + * Support non-equals conditions in lightweight transactions (CASSANDRA-6839) + * Add IF [NOT] EXISTS to create/drop triggers (CASSANDRA-7606) + * (cqlsh) Display the current logged-in user (CASSANDRA-7785) + * (cqlsh) Don't ignore CTRL-C during COPY FROM execution (CASSANDRA-7815) + * (cqlsh) Order UDTs according to cross-type dependencies in DESCRIBE + output (CASSANDRA-7659) + * (cqlsh) Fix handling of CAS statement results (CASSANDRA-7671) + * (cqlsh) COPY TO/FROM improvements (CASSANDRA-7405) + * Support list index operations with conditions (CASSANDRA-7499) + * Add max live/tombstoned cells to nodetool cfstats output (CASSANDRA-7731) + * Validate IPv6 wildcard addresses properly (CASSANDRA-7680) + * (cqlsh) Error when tracing query (CASSANDRA-7613) + * Avoid IOOBE when building SyntaxError message snippet (CASSANDRA-7569) + * SSTableExport uses correct validator to create string representation of partition + keys (CASSANDRA-7498) + * Avoid NPEs when receiving type changes for an unknown keyspace (CASSANDRA-7689) + * Add support for custom 2i validation (CASSANDRA-7575) + * Pig support for hadoop CqlInputFormat (CASSANDRA-6454) + * Add listen_interface and rpc_interface options (CASSANDRA-7417) + * Improve schema merge performance (CASSANDRA-7444) + * Adjust MT depth based on # of partition validating (CASSANDRA-5263) + * Optimise NativeCell comparisons (CASSANDRA-6755) + * Configurable client timeout for cqlsh (CASSANDRA-7516) + * Include snippet of CQL query near syntax error in messages (CASSANDRA-7111) +Merged from 2.0: + * Add metrics for prepared statement usage and eviction (CASSANDRA-7930) * Make CQLSSTableWriter sync within partitions (CASSANDRA-7360) * Potentially use non-local replicas in CqlConfigHelper (CASSANDRA-7906) - * Explicitly disallowing mixing multi-column and single-column + * Explicitly disallow mixing multi-column and single-column relations on clustering columns (CASSANDRA-7711) * Better error message when condition is set on PK column (CASSANDRA-7804) + * Don't send schema change responses and events for no-op DDL + statements (CASSANDRA-7600) + * (Hadoop) fix cluster initialisation for a split fetching (CASSANDRA-7774) + * Throw InvalidRequestException when queries contain relations on entire + collection columns (CASSANDRA-7506) + * (cqlsh) enable CTRL-R history search with libedit (CASSANDRA-7577) + * (Hadoop) allow ACFRW to limit nodes to local DC (CASSANDRA-7252) + * (cqlsh) cqlsh should automatically disable tracing when selecting + from system_traces (CASSANDRA-7641) + * (Hadoop) Add CqlOutputFormat (CASSANDRA-6927) + * Don't depend on cassandra config for nodetool ring (CASSANDRA-7508) + * (cqlsh) Fix failing cqlsh formatting tests (CASSANDRA-7703) + * Fix IncompatibleClassChangeError from hadoop2 (CASSANDRA-7229) + * Add 'nodetool sethintedhandoffthrottlekb' (CASSANDRA-7635) + * (cqlsh) Add tab-completion for CREATE/DROP USER IF [NOT] EXISTS (CASSANDRA-7611) + * Catch errors when the JVM pulls the rug out from GCInspector (CASSANDRA-5345) + * cqlsh fails when version number parts are not int (CASSANDRA-7524) +Merged from 1.2: + * Don't index tombstones (CASSANDRA-7828) + * Improve PasswordAuthenticator default super user setup (CASSANDRA-7788) + + +2.1.0 + * (cqlsh) Removed "ALTER TYPE RENAME TO " from tab-completion + (CASSANDRA-7895) + * Fixed IllegalStateException in anticompaction (CASSANDRA-7892) + * cqlsh: DESCRIBE support for frozen UDTs, tuples (CASSANDRA-7863) + * Avoid exposing internal classes over JMX (CASSANDRA-7879) + * Add null check for keys when freezing collection (CASSANDRA-7869) + * Improve stress workload realism (CASSANDRA-7519) + + +2.1.0-rc7 + * Add frozen keyword and require UDT to be frozen (CASSANDRA-7857) + * Track added sstable size correctly (CASSANDRA-7239) + * (cqlsh) Fix case insensitivity (CASSANDRA-7834) + * Fix failure to stream ranges when moving (CASSANDRA-7836) + * Correctly remove tmplink files (CASSANDRA-7803) + * (cqlsh) Fix column name formatting for functions, CAS operations, + and UDT field selections (CASSANDRA-7806) + * (cqlsh) Fix COPY FROM handling of null/empty primary key + values (CASSANDRA-7792) + * Fix ordering of static cells (CASSANDRA-7763) +Merged from 2.0: * Forbid re-adding dropped counter columns (CASSANDRA-7831) * Fix CFMetaData#isThriftCompatible() for PK-only tables (CASSANDRA-7832) * Always reject inequality on the partition key without token() http://git-wip-us.apache.org/repos/asf/cassandra/blob/681c380b/src/java/org/apache/cassandra/cql3/QueryProcessor.java ---------------------------------------------------------------------- diff --cc src/java/org/apache/cassandra/cql3/QueryProcessor.java index 99972a2,ee188a3..c983d72 --- a/src/java/org/apache/cassandra/cql3/QueryProcessor.java +++ b/src/java/org/apache/cassandra/cql3/QueryProcessor.java @@@ -19,10 -19,9 +19,12 @@@ package org.apache.cassandra.cql3 import java.nio.ByteBuffer; import java.util.*; +import java.util.concurrent.ConcurrentMap; +import java.util.concurrent.ConcurrentHashMap; + import java.util.concurrent.TimeUnit; + import java.util.concurrent.atomic.AtomicLong; +import com.google.common.annotations.VisibleForTesting; import com.google.common.primitives.Ints; import com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap; @@@ -36,13 -34,11 +38,15 @@@ import org.slf4j.LoggerFactory import org.apache.cassandra.cql3.statements.*; import org.apache.cassandra.db.*; +import org.apache.cassandra.db.composites.*; +import org.apache.cassandra.db.marshal.AbstractType; import org.apache.cassandra.exceptions.*; + import org.apache.cassandra.metrics.CqlStatementMetrics; import org.apache.cassandra.service.ClientState; import org.apache.cassandra.service.QueryState; +import org.apache.cassandra.service.pager.QueryPager; +import org.apache.cassandra.service.pager.QueryPagers; + import org.apache.cassandra.service.StorageService; import org.apache.cassandra.thrift.ThriftClientState; import org.apache.cassandra.tracing.Tracing; import org.apache.cassandra.transport.messages.ResultMessage; @@@ -78,72 -75,67 +82,87 @@@ public class QueryProcessor implements } }; - private static final ConcurrentLinkedHashMap preparedStatements; + private static final ConcurrentLinkedHashMap preparedStatements; private static final ConcurrentLinkedHashMap thriftPreparedStatements; + // A map for prepared statements used internally (which we don't want to mix with user statement, in particular we don't + // bother with expiration on those. + private static final ConcurrentMap internalStatements = new ConcurrentHashMap<>(); + + @VisibleForTesting public static final CqlStatementMetrics metrics = new CqlStatementMetrics(); + private static AtomicLong evictionCount = new AtomicLong(0); static { - if (MemoryMeter.isInitialized()) - { - preparedStatements = new ConcurrentLinkedHashMap.Builder() - .maximumWeightedCapacity(MAX_CACHE_PREPARED_MEMORY) - .weigher(cqlMemoryUsageWeigher) - .listener(new EvictionListener() - { - @Override - public void onEviction(MD5Digest md5Digest, CQLStatement prepared) - { - metrics.activePreparedStatements.dec(); - metrics.evictedPreparedStatements.inc(); - evictionCount.incrementAndGet(); - } - }).build(); - thriftPreparedStatements = new ConcurrentLinkedHashMap.Builder() - .maximumWeightedCapacity(MAX_CACHE_PREPARED_MEMORY) - .weigher(thriftMemoryUsageWeigher) - .listener(new EvictionListener() - { - @Override - public void onEviction(Integer i, CQLStatement prepared) - { - metrics.activePreparedStatements.dec(); - metrics.evictedPreparedStatements.inc(); - evictionCount.incrementAndGet(); - } - }).build(); - } - else - { - logger.error("Unable to initialize MemoryMeter (jamm not specified as javaagent). This means " - + "Cassandra will be unable to measure object sizes accurately and may consequently OOM."); - preparedStatements = new ConcurrentLinkedHashMap.Builder() - .maximumWeightedCapacity(MAX_CACHE_PREPARED_COUNT) - .build(); - thriftPreparedStatements = new ConcurrentLinkedHashMap.Builder() - .maximumWeightedCapacity(MAX_CACHE_PREPARED_COUNT) - .build(); - } + preparedStatements = new ConcurrentLinkedHashMap.Builder() + .maximumWeightedCapacity(MAX_CACHE_PREPARED_MEMORY) + .weigher(cqlMemoryUsageWeigher) + .listener(new EvictionListener() + { + @Override + public void onEviction(MD5Digest md5Digest, ParsedStatement.Prepared prepared) + { + metrics.activePreparedStatements.dec(); ++ metrics.evictedPreparedStatements.inc(); ++ evictionCount.incrementAndGet(); + } + }).build(); + + thriftPreparedStatements = new ConcurrentLinkedHashMap.Builder() + .maximumWeightedCapacity(MAX_CACHE_PREPARED_MEMORY) + .weigher(thriftMemoryUsageWeigher) + .listener(new EvictionListener() + { + @Override + public void onEviction(Integer integer, CQLStatement cqlStatement) + { + metrics.activePreparedStatements.dec(); ++ metrics.evictedPreparedStatements.inc(); ++ evictionCount.incrementAndGet(); + } + }) + .build(); + StorageService.scheduledTasks.scheduleAtFixedRate(new Runnable() { + @Override + public void run() { + long count = evictionCount.getAndSet(0); + if (count > 0) + { + logger.info("{} prepared statements discarded in the last minute because cache limit reached (cache limit = {} bytes)", count, MAX_CACHE_PREPARED_MEMORY); + } + } + }, 1, 1, TimeUnit.MINUTES); } + // Work aound initialization dependency + private static enum InternalStateInstance + { + INSTANCE; + + private final QueryState queryState; + + InternalStateInstance() + { + ClientState state = ClientState.forInternalCalls(); + try + { + state.setKeyspace(Keyspace.SYSTEM_KS); + } + catch (InvalidRequestException e) + { + throw new RuntimeException(); + } + this.queryState = new QueryState(state); + } + } + + private static QueryState internalQueryState() + { + return InternalStateInstance.INSTANCE.queryState; + } + private QueryProcessor() { } @@@ -394,28 -306,26 +413,24 @@@ throw new InvalidRequestException(String.format("Prepared statement of size %d bytes is larger than allowed maximum of %d bytes.", statementSize, MAX_CACHE_PREPARED_MEMORY)); - try - + if (forThrift) { - if (forThrift) - { - int statementId = toHash.hashCode(); - thriftPreparedStatements.put(statementId, prepared.statement); - logger.trace("Stored prepared statement #{} with {} bind markers", - statementId, - prepared.statement.getBoundTerms()); - return ResultMessage.Prepared.forThrift(statementId, prepared.boundNames); - } else - { - MD5Digest statementId = MD5Digest.compute(toHash); - preparedStatements.put(statementId, prepared); - logger.trace("Stored prepared statement #{} with {} bind markers", - statementId, - prepared.statement.getBoundTerms()); - return new ResultMessage.Prepared(statementId, prepared); - } - } finally + int statementId = toHash.hashCode(); + thriftPreparedStatements.put(statementId, prepared.statement); + metrics.activePreparedStatements.inc(); - logger.trace(String.format("Stored prepared statement #%d with %d bind markers", - statementId, - prepared.statement.getBoundTerms())); ++ logger.trace("Stored prepared statement #{} with {} bind markers", ++ statementId, ++ prepared.statement.getBoundTerms()); + return ResultMessage.Prepared.forThrift(statementId, prepared.boundNames); - } - else ++ } else { + MD5Digest statementId = MD5Digest.compute(toHash); - preparedStatements.put(statementId, prepared.statement); ++ preparedStatements.put(statementId, prepared); metrics.activePreparedStatements.inc(); - logger.trace(String.format("Stored prepared statement %s with %d bind markers", - statementId, - prepared.statement.getBoundTerms())); ++ logger.trace("Stored prepared statement #{} with {} bind markers", ++ statementId, ++ prepared.statement.getBoundTerms()); + return new ResultMessage.Prepared(statementId, prepared); } }