Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 13205D101 for ; Fri, 27 Jul 2012 08:14:48 +0000 (UTC) Received: (qmail 52507 invoked by uid 500); 27 Jul 2012 08:14:47 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 52317 invoked by uid 500); 27 Jul 2012 08:14:47 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 52279 invoked by uid 99); 27 Jul 2012 08:14:46 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Jul 2012 08:14:46 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id 9947219438; Fri, 27 Jul 2012 08:14:46 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: slebresne@apache.org To: commits@cassandra.apache.org X-Mailer: ASF-Git Admin Mailer Subject: [1/2] git commit: Merge branch 'cassandra-1.0' into cassandra-1.1 Message-Id: <20120727081446.9947219438@tyr.zones.apache.org> Date: Fri, 27 Jul 2012 08:14:46 +0000 (UTC) Updated Branches: refs/heads/cassandra-1.1 070c1d8dd -> ccd436c0f Merge branch 'cassandra-1.0' into cassandra-1.1 Conflicts: CHANGES.txt Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ccd436c0 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ccd436c0 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ccd436c0 Branch: refs/heads/cassandra-1.1 Commit: ccd436c0fe0c75db09dd7cd87a516f005f4e7289 Parents: 070c1d8 7ff8e3c Author: Sylvain Lebresne Authored: Fri Jul 27 10:14:34 2012 +0200 Committer: Sylvain Lebresne Committed: Fri Jul 27 10:14:34 2012 +0200 ---------------------------------------------------------------------- CHANGES.txt | 2 ++ .../cassandra/db/compaction/CompactionManager.java | 9 ++++++--- 2 files changed, 8 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ccd436c0/CHANGES.txt ---------------------------------------------------------------------- diff --cc CHANGES.txt index bcf56b7,b3fa1a8..fd6991c --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,155 -1,6 +1,157 @@@ -1.0.11 +1.1.3 + * (JMX) rename getRangeKeySample to sampleKeyRange to avoid returning + multi-MB results as an attribute (CASSANDRA-4452) + * flush based on data size, not throughput; overwritten columns no + longer artificially inflate liveRatio (CASSANDRA-4399) + * update default commitlog segment size to 32MB and total commitlog + size to 32/1024 MB for 32/64 bit JVMs, respectively (CASSANDRA-4422) + * avoid using global partitioner to estimate ranges in index sstables + (CASSANDRA-4403) + * restore pre-CASSANDRA-3862 approach to removing expired tombstones + from row cache during compaction (CASSANDRA-4364) + * (stress) support for CQL prepared statements (CASSANDRA-3633) + * Correctly catch exception when Snappy cannot be loaded (CASSANDRA-4400) + * (cql3) Support ORDER BY when IN condition is given in WHERE clause (CASSANDRA-4327) + * (cql3) delete "component_index" column on DROP TABLE call (CASSANDRA-4420) + * change nanoTime() to currentTimeInMillis() in schema related code (CASSANDRA-4432) + * add a token generation tool (CASSANDRA-3709) + * Fix LCS bug with sstable containing only 1 row (CASSANDRA-4411) + * fix "Can't Modify Index Name" problem on CF update (CASSANDRA-4439) + * Fix assertion error in getOverlappingSSTables during repair (CASSANDRA-4456) + * fix nodetool's setcompactionthreshold command (CASSANDRA-4455) + * Ensure compacted files are never used, to avoid counter overcount (CASSANDRA-4436) +Merged from 1.0: + * Push the validation of secondary index values to the SecondaryIndexManager (CASSANDRA-4240) * allow dropping columns shadowed by not-yet-expired supercolumn or row tombstones in PrecompactedRow (CASSANDRA-4396) + * fix 1.0.x node join to mixed version cluster, other nodes >= 1.1 (CASSANDRA-4195) + * Fix LCS splitting sstable base on uncompressed size (CASSANDRA-4419) + * Bootstraps that fail are detected upon restart and will retry safely without + needing to delete existing data first (CASSANDRA-4427) + * (cqlsh) add a COPY TO command to copy a CF to a CSV file (CASSANDRA-4434) ++ * Don't purge columns during upgradesstables (CASSANDRA-4462) ++ * Push the validation of secondary index values to the SecondaryIndexManager (CASSANDRA-4240) + + +1.1.2 + * Fix cleanup not deleting index entries (CASSANDRA-4379) + * Use correct partitioner when saving + loading caches (CASSANDRA-4331) + * Check schema before trying to export sstable (CASSANDRA-2760) + * Raise a meaningful exception instead of NPE when PFS encounters + an unconfigured node + no default (CASSANDRA-4349) + * fix bug in sstable blacklisting with LCS (CASSANDRA-4343) + * LCS no longer promotes tiny sstables out of L0 (CASSANDRA-4341) + * skip tombstones during hint replay (CASSANDRA-4320) + * fix NPE in compactionstats (CASSANDRA-4318) + * enforce 1m min keycache for auto (CASSANDRA-4306) + * Have DeletedColumn.isMFD always return true (CASSANDRA-4307) + * (cql3) exeption message for ORDER BY constraints said primary filter can be + an IN clause, which is misleading (CASSANDRA-4319) + * (cql3) Reject (not yet supported) creation of 2ndardy indexes on tables with + composite primary keys (CASSANDRA-4328) + * Set JVM stack size to 160k for java 7 (CASSANDRA-4275) + * cqlsh: add COPY command to load data from CSV flat files (CASSANDRA-4012) + * CFMetaData.fromThrift to throw ConfigurationException upon error (CASSANDRA-4353) + * Use CF comparator to sort indexed columns in SecondaryIndexManager + (CASSANDRA-4365) + * add strategy_options to the KSMetaData.toString() output (CASSANDRA-4248) + * (cql3) fix range queries containing unqueried results (CASSANDRA-4372) + * (cql3) allow updating column_alias types (CASSANDRA-4041) + * (cql3) Fix deletion bug (CASSANDRA-4193) + * Fix computation of overlapping sstable for leveled compaction (CASSANDRA-4321) + * Improve scrub and allow to run it offline (CASSANDRA-4321) + * Fix assertionError in StorageService.bulkLoad (CASSANDRA-4368) + * (cqlsh) add option to authenticate to a keyspace at startup (CASSANDRA-4108) + * (cqlsh) fix ASSUME functionality (CASSANDRA-4352) + * Fix ColumnFamilyRecordReader to not return progress > 100% (CASSANDRA-3942) +Merged from 1.0: + * Set gc_grace on index CF to 0 (CASSANDRA-4314) + + +1.1.1 + * allow larger cache capacities than 2GB (CASSANDRA-4150) + * add getsstables command to nodetool (CASSANDRA-4199) + * apply parent CF compaction settings to secondary index CFs (CASSANDRA-4280) + * preserve commitlog size cap when recycling segments at startup + (CASSANDRA-4201) + * (Hadoop) fix split generation regression (CASSANDRA-4259) + * ignore min/max compactions settings in LCS, while preserving + behavior that min=max=0 disables autocompaction (CASSANDRA-4233) + * log number of rows read from saved cache (CASSANDRA-4249) + * calculate exact size required for cleanup operations (CASSANDRA-1404) + * avoid blocking additional writes during flush when the commitlog + gets behind temporarily (CASSANDRA-1991) + * enable caching on index CFs based on data CF cache setting (CASSANDRA-4197) + * warn on invalid replication strategy creation options (CASSANDRA-4046) + * remove [Freeable]Memory finalizers (CASSANDRA-4222) + * include tombstone size in ColumnFamily.size, which can prevent OOM + during sudden mass delete operations by yielding a nonzero liveRatio + (CASSANDRA-3741) + * Open 1 sstableScanner per level for leveled compaction (CASSANDRA-4142) + * Optimize reads when row deletion timestamps allow us to restrict + the set of sstables we check (CASSANDRA-4116) + * add support for commitlog archiving and point-in-time recovery + (CASSANDRA-3690) + * avoid generating redundant compaction tasks during streaming + (CASSANDRA-4174) + * add -cf option to nodetool snapshot, and takeColumnFamilySnapshot to + StorageService mbean (CASSANDRA-556) + * optimize cleanup to drop entire sstables where possible (CASSANDRA-4079) + * optimize truncate when autosnapshot is disabled (CASSANDRA-4153) + * update caches to use byte[] keys to reduce memory overhead (CASSANDRA-3966) + * add column limit to cli (CASSANDRA-3012, 4098) + * clean up and optimize DataOutputBuffer, used by CQL compression and + CompositeType (CASSANDRA-4072) + * optimize commitlog checksumming (CASSANDRA-3610) + * identify and blacklist corrupted SSTables from future compactions + (CASSANDRA-2261) + * Move CfDef and KsDef validation out of thrift (CASSANDRA-4037) + * Expose API to repair a user provided range (CASSANDRA-3912) + * Add way to force the cassandra-cli to refresh its schema (CASSANDRA-4052) + * Avoid having replicate on write tasks stacking up at CL.ONE (CASSANDRA-2889) + * (cql3) Backwards compatibility for composite comparators in non-cql3-aware + clients (CASSANDRA-4093) + * (cql3) Fix order by for reversed queries (CASSANDRA-4160) + * (cql3) Add ReversedType support (CASSANDRA-4004) + * (cql3) Add timeuuid type (CASSANDRA-4194) + * (cql3) Minor fixes (CASSANDRA-4185) + * (cql3) Fix prepared statement in BATCH (CASSANDRA-4202) + * (cql3) Reduce the list of reserved keywords (CASSANDRA-4186) + * (cql3) Move max/min compaction thresholds to compaction strategy options + (CASSANDRA-4187) + * Fix exception during move when localhost is the only source (CASSANDRA-4200) + * (cql3) Allow paging through non-ordered partitioner results (CASSANDRA-3771) + * (cql3) Fix drop index (CASSANDRA-4192) + * (cql3) Don't return range ghosts anymore (CASSANDRA-3982) + * fix re-creating Keyspaces/ColumnFamilies with the same name as dropped + ones (CASSANDRA-4219) + * fix SecondaryIndex LeveledManifest save upon snapshot (CASSANDRA-4230) + * fix missing arrayOffset in FBUtilities.hash (CASSANDRA-4250) + * (cql3) Add name of parameters in CqlResultSet (CASSANDRA-4242) + * (cql3) Correctly validate order by queries (CASSANDRA-4246) + * rename stress to cassandra-stress for saner packaging (CASSANDRA-4256) + * Fix exception on colum metadata with non-string comparator (CASSANDRA-4269) + * Check for unknown/invalid compression options (CASSANDRA-4266) + * (cql3) Adds simple access to column timestamp and ttl (CASSANDRA-4217) + * (cql3) Fix range queries with secondary indexes (CASSANDRA-4257) + * Better error messages from improper input in cli (CASSANDRA-3865) + * Try to stop all compaction upon Keyspace or ColumnFamily drop (CASSANDRA-4221) + * (cql3) Allow keyspace properties to contain hyphens (CASSANDRA-4278) + * (cql3) Correctly validate keyspace access in create table (CASSANDRA-4296) + * Avoid deadlock in migration stage (CASSANDRA-3882) + * Take supercolumn names and deletion info into account in memtable throughput + (CASSANDRA-4264) + * Add back backward compatibility for old style replication factor (CASSANDRA-4294) + * Preserve compatibility with pre-1.1 index queries (CASSANDRA-4262) +Merged from 1.0: + * Fix super columns bug where cache is not updated (CASSANDRA-4190) + * fix maxTimestamp to include row tombstones (CASSANDRA-4116) + * (CLI) properly handle quotes in create/update keyspace commands (CASSANDRA-4129) + * Avoids possible deadlock during bootstrap (CASSANDRA-4159) + * fix stress tool that hangs forever on timeout or error (CASSANDRA-4128) + * stress tool to return appropriate exit code on failure (CASSANDRA-4188) + * fix compaction NPE when out of disk space and assertions disabled + (CASSANDRA-3985) * synchronize LCS getEstimatedTasks to avoid CME (CASSANDRA-4255) * ensure unique streaming session id's (CASSANDRA-4223) * kick off background compaction when min/max thresholds change http://git-wip-us.apache.org/repos/asf/cassandra/blob/ccd436c0/src/java/org/apache/cassandra/db/compaction/CompactionManager.java ---------------------------------------------------------------------- diff --cc src/java/org/apache/cassandra/db/compaction/CompactionManager.java index 4b39280,2400cfe..0f5be39 --- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java +++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java @@@ -72,14 -67,18 +72,17 @@@ public class CompactionManager implemen private static final Logger logger = LoggerFactory.getLogger(CompactionManager.class); public static final CompactionManager instance; + public static final int NO_GC = Integer.MIN_VALUE; + public static final int GC_ALL = Integer.MAX_VALUE; + /** * compactionLock has two purposes: - * - Compaction acquires its readLock so that multiple compactions can happen simultaneously, - * but the KS/CF migtations acquire its writeLock, so they can be sure no new SSTables will - * be created for a dropped CF posthumously. (Thus, compaction checks CFS.isValid while the - * lock is acquired.) * - "Special" compactions will acquire writelock instead of readlock to make sure that all - * other compaction activity is quiesced and they can grab ALL the sstables to do something. - * TODO this is too big a hammer -- we should only care about quiescing all for the given CFS. + * other compaction activity is quiesced and they can grab ALL the sstables to do something. + * - Some schema migrations cannot run concurrently with compaction. (Currently, this is + * only when changing compaction strategy -- see CFS.maybeReloadCompactionStrategy.) + * + * TODO this is too big a hammer -- we should only care about quiescing all for the given CFS. */ private final ReentrantReadWriteLock compactionLock = new ReentrantReadWriteLock(); @@@ -830,35 -983,19 +833,35 @@@ private static class ValidationCompactionIterable extends CompactionIterable { - public ValidationCompactionIterable(ColumnFamilyStore cfs, Collection sstables, Range range) throws IOException + public ValidationCompactionIterable(ColumnFamilyStore cfs, Collection sstables, Range range) throws IOException { super(OperationType.VALIDATION, - getScanners(sstables, range), - new CompactionController(cfs, sstables, getDefaultGcBefore(cfs), true)); + cfs.getCompactionStrategy().getScanners(sstables, range), + new ValidationCompactionController(cfs, sstables)); } + } - protected static List getScanners(Iterable sstables, Range range) throws IOException + /* + * Controller for validation compaction that never purges. + * Note that we should not call cfs.getOverlappingSSTables on the provided + * sstables because those sstables are not guaranteed to be active sstables + * (since we can run repair on a snapshot). + */ + private static class ValidationCompactionController extends CompactionController + { + public ValidationCompactionController(ColumnFamilyStore cfs, Collection sstables) { - ArrayList scanners = new ArrayList(); - for (SSTableReader sstable : sstables) - scanners.add(sstable.getDirectScanner(range)); - return scanners; + super(cfs, - Integer.MAX_VALUE, ++ NO_GC, + true, + null, + cfs.getCompactionStrategy().isKeyExistenceExpensive(ImmutableSet.copyOf(sstables))); + } + + @Override + public boolean shouldPurge(DecoratedKey key) + { + return false; } }