Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7410F179E0 for ; Wed, 28 Jan 2015 15:46:14 +0000 (UTC) Received: (qmail 63162 invoked by uid 500); 28 Jan 2015 15:46:14 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 63043 invoked by uid 500); 28 Jan 2015 15:46:14 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 63025 invoked by uid 99); 28 Jan 2015 15:46:14 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jan 2015 15:46:14 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id A1210E07D9; Wed, 28 Jan 2015 15:46:14 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: aleksey@apache.org To: commits@cassandra.apache.org Date: Wed, 28 Jan 2015 15:46:15 -0000 Message-Id: <934b5f6c52164a19a69fc61ed03e609d@git.apache.org> In-Reply-To: <50c73ace7d66489f90de01a1c70559da@git.apache.org> References: <50c73ace7d66489f90de01a1c70559da@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [2/2] cassandra git commit: Merge branch 'cassandra-2.1' into trunk Merge branch 'cassandra-2.1' into trunk Conflicts: CHANGES.txt src/java/org/apache/cassandra/config/ColumnDefinition.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bbb1592e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bbb1592e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bbb1592e Branch: refs/heads/trunk Commit: bbb1592e84d2949de20b8e1699ee12f027815a88 Parents: 9c4a776 b788bb3 Author: Aleksey Yeschenko Authored: Wed Jan 28 18:46:05 2015 +0300 Committer: Aleksey Yeschenko Committed: Wed Jan 28 18:46:05 2015 +0300 ---------------------------------------------------------------------- CHANGES.txt | 7 +++++-- src/java/org/apache/cassandra/thrift/ThriftConversion.java | 2 +- 2 files changed, 6 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/cassandra/blob/bbb1592e/CHANGES.txt ---------------------------------------------------------------------- diff --cc CHANGES.txt index 53a2c63,b976a16..36b081b --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,62 -1,5 +1,61 @@@ +3.0 - * rpc_interface and listen_interface generate NPE on startup when specified interface doesn't exist (CASSANDRA-8677) - * Fix ArrayIndexOutOfBoundsException in nodetool cfhistograms (CASSANDRA-8514) + * Serializing Row cache alternative, fully off heap (CASSANDRA-7438) + * Duplicate rows returned when in clause has repeated values (CASSANDRA-6707) + * Make CassandraException unchecked, extend RuntimeException (CASSANDRA-8560) + * Support direct buffer decompression for reads (CASSANDRA-8464) + * DirectByteBuffer compatible LZ4 methods (CASSANDRA-7039) + * Add role based access control (CASSANDRA-7653) + * Group sstables for anticompaction correctly (CASSANDRA-8578) + * Add ReadFailureException to native protocol, respond + immediately when replicas encounter errors while handling + a read request (CASSANDRA-7886) + * Switch CommitLogSegment from RandomAccessFile to nio (CASSANDRA-8308) + * Allow mixing token and partition key restrictions (CASSANDRA-7016) + * Support index key/value entries on map collections (CASSANDRA-8473) + * Modernize schema tables (CASSANDRA-8261) + * Support for user-defined aggregation functions (CASSANDRA-8053) + * Fix NPE in SelectStatement with empty IN values (CASSANDRA-8419) + * Refactor SelectStatement, return IN results in natural order instead + of IN value list order and ignore duplicate values in partition key IN restrictions (CASSANDRA-7981) + * Support UDTs, tuples, and collections in user-defined + functions (CASSANDRA-7563) + * Fix aggregate fn results on empty selection, result column name, + and cqlsh parsing (CASSANDRA-8229) + * Mark sstables as repaired after full repair (CASSANDRA-7586) + * Extend Descriptor to include a format value and refactor reader/writer + APIs (CASSANDRA-7443) + * Integrate JMH for microbenchmarks (CASSANDRA-8151) + * Keep sstable levels when bootstrapping (CASSANDRA-7460) + * Add Sigar library and perform basic OS settings check on startup (CASSANDRA-7838) + * Support for aggregation functions (CASSANDRA-4914) + * Remove cassandra-cli (CASSANDRA-7920) + * Accept dollar quoted strings in CQL (CASSANDRA-7769) + * Make assassinate a first class command (CASSANDRA-7935) + * Support IN clause on any clustering column (CASSANDRA-4762) + * Improve compaction logging (CASSANDRA-7818) + * Remove YamlFileNetworkTopologySnitch (CASSANDRA-7917) + * Do anticompaction in groups (CASSANDRA-6851) + * Support user-defined functions (CASSANDRA-7395, 7526, 7562, 7740, 7781, 7929, + 7924, 7812, 8063, 7813, 7708) + * Permit configurable timestamps with cassandra-stress (CASSANDRA-7416) + * Move sstable RandomAccessReader to nio2, which allows using the + FILE_SHARE_DELETE flag on Windows (CASSANDRA-4050) + * Remove CQL2 (CASSANDRA-5918) + * Add Thrift get_multi_slice call (CASSANDRA-6757) + * Optimize fetching multiple cells by name (CASSANDRA-6933) + * Allow compilation in java 8 (CASSANDRA-7028) + * Make incremental repair default (CASSANDRA-7250) + * Enable code coverage thru JaCoCo (CASSANDRA-7226) + * Switch external naming of 'column families' to 'tables' (CASSANDRA-4369) + * Shorten SSTable path (CASSANDRA-6962) + * Use unsafe mutations for most unit tests (CASSANDRA-6969) + * Fix race condition during calculation of pending ranges (CASSANDRA-7390) + * Fail on very large batch sizes (CASSANDRA-8011) + * Improve concurrency of repair (CASSANDRA-6455, 8208) + + 2.1.3 + * Fix schema from Thrift conversion with empty metadata (CASSANDRA-8695) * Safer Resource Management (CASSANDRA-7705) * Make sure we compact highly overlapping cold sstables with STCS (CASSANDRA-8635) http://git-wip-us.apache.org/repos/asf/cassandra/blob/bbb1592e/src/java/org/apache/cassandra/thrift/ThriftConversion.java ---------------------------------------------------------------------- diff --cc src/java/org/apache/cassandra/thrift/ThriftConversion.java index 066ddb8,2aca45a..adb925e --- a/src/java/org/apache/cassandra/thrift/ThriftConversion.java +++ b/src/java/org/apache/cassandra/thrift/ThriftConversion.java @@@ -153,337 -130,4 +153,337 @@@ public class ThriftConversio } return converted; } + + public static KSMetaData fromThrift(KsDef ksd, CFMetaData... cfDefs) throws ConfigurationException + { + Class cls = AbstractReplicationStrategy.getClass(ksd.strategy_class); + if (cls.equals(LocalStrategy.class)) + throw new ConfigurationException("Unable to use given strategy class: LocalStrategy is reserved for internal use."); + + return new KSMetaData(ksd.name, + cls, + ksd.strategy_options == null ? Collections.emptyMap() : ksd.strategy_options, + ksd.durable_writes, + Arrays.asList(cfDefs)); + } + + public static KsDef toThrift(KSMetaData ksm) + { + List cfDefs = new ArrayList<>(ksm.cfMetaData().size()); + for (CFMetaData cfm : ksm.cfMetaData().values()) + if (cfm.isThriftCompatible()) // Don't expose CF that cannot be correctly handle by thrift; see CASSANDRA-4377 for further details + cfDefs.add(toThrift(cfm)); + + KsDef ksdef = new KsDef(ksm.name, ksm.strategyClass.getName(), cfDefs); + ksdef.setStrategy_options(ksm.strategyOptions); + ksdef.setDurable_writes(ksm.durableWrites); + + return ksdef; + } + + public static CFMetaData fromThrift(CfDef cf_def) + throws org.apache.cassandra.exceptions.InvalidRequestException, ConfigurationException + { + return internalFromThrift(cf_def, Collections.emptyList()); + } + + public static CFMetaData fromThriftForUpdate(CfDef cf_def, CFMetaData toUpdate) + throws org.apache.cassandra.exceptions.InvalidRequestException, ConfigurationException + { + return internalFromThrift(cf_def, toUpdate.allColumns()); + } + + // Convert a thrift CfDef, given a list of ColumnDefinitions to copy over to the created CFMetadata before the CQL metadata are rebuild + private static CFMetaData internalFromThrift(CfDef cf_def, Collection previousCQLMetadata) + throws org.apache.cassandra.exceptions.InvalidRequestException, ConfigurationException + { + ColumnFamilyType cfType = ColumnFamilyType.create(cf_def.column_type); + if (cfType == null) + throw new org.apache.cassandra.exceptions.InvalidRequestException("Invalid column type " + cf_def.column_type); + + applyImplicitDefaults(cf_def); + + try + { + AbstractType rawComparator = TypeParser.parse(cf_def.comparator_type); + AbstractType subComparator = cfType == ColumnFamilyType.Standard + ? null + : cf_def.subcomparator_type == null ? BytesType.instance : TypeParser.parse(cf_def.subcomparator_type); + + AbstractType fullRawComparator = CFMetaData.makeRawAbstractType(rawComparator, subComparator); + + AbstractType keyValidator = cf_def.isSetKey_validation_class() ? TypeParser.parse(cf_def.key_validation_class) : null; + + // Convert the REGULAR definitions from the input CfDef + List defs = fromThrift(cf_def.keyspace, cf_def.name, rawComparator, subComparator, cf_def.column_metadata); + + // Add the keyAlias if there is one, since that's on CQL metadata that thrift can actually change (for + // historical reasons) + boolean hasKeyAlias = cf_def.isSetKey_alias() && keyValidator != null && !(keyValidator instanceof CompositeType); + if (hasKeyAlias) + defs.add(ColumnDefinition.partitionKeyDef(cf_def.keyspace, cf_def.name, cf_def.key_alias, keyValidator, null)); + + // Now add any CQL metadata that we want to copy, skipping the keyAlias if there was one + for (ColumnDefinition def : previousCQLMetadata) + { + // isPartOfCellName basically means 'is not just a CQL metadata' + if (def.isPartOfCellName()) + continue; + + if (def.kind == ColumnDefinition.Kind.PARTITION_KEY && hasKeyAlias) + continue; + + defs.add(def); + } + + CellNameType comparator = CellNames.fromAbstractType(fullRawComparator, CFMetaData.calculateIsDense(fullRawComparator, defs)); + + UUID cfId = Schema.instance.getId(cf_def.keyspace, cf_def.name); + if (cfId == null) + cfId = UUIDGen.getTimeUUID(); + + CFMetaData newCFMD = new CFMetaData(cf_def.keyspace, cf_def.name, cfType, comparator, cfId); + + newCFMD.addAllColumnDefinitions(defs); + + if (keyValidator != null) + newCFMD.keyValidator(keyValidator); + if (cf_def.isSetGc_grace_seconds()) + newCFMD.gcGraceSeconds(cf_def.gc_grace_seconds); + if (cf_def.isSetMin_compaction_threshold()) + newCFMD.minCompactionThreshold(cf_def.min_compaction_threshold); + if (cf_def.isSetMax_compaction_threshold()) + newCFMD.maxCompactionThreshold(cf_def.max_compaction_threshold); + if (cf_def.isSetCompaction_strategy()) + newCFMD.compactionStrategyClass(CFMetaData.createCompactionStrategy(cf_def.compaction_strategy)); + if (cf_def.isSetCompaction_strategy_options()) + newCFMD.compactionStrategyOptions(new HashMap<>(cf_def.compaction_strategy_options)); + if (cf_def.isSetBloom_filter_fp_chance()) + newCFMD.bloomFilterFpChance(cf_def.bloom_filter_fp_chance); + if (cf_def.isSetMemtable_flush_period_in_ms()) + newCFMD.memtableFlushPeriod(cf_def.memtable_flush_period_in_ms); + if (cf_def.isSetCaching() || cf_def.isSetCells_per_row_to_cache()) + newCFMD.caching(CachingOptions.fromThrift(cf_def.caching, cf_def.cells_per_row_to_cache)); + if (cf_def.isSetRead_repair_chance()) + newCFMD.readRepairChance(cf_def.read_repair_chance); + if (cf_def.isSetDefault_time_to_live()) + newCFMD.defaultTimeToLive(cf_def.default_time_to_live); + if (cf_def.isSetDclocal_read_repair_chance()) + newCFMD.dcLocalReadRepairChance(cf_def.dclocal_read_repair_chance); + if (cf_def.isSetMin_index_interval()) + newCFMD.minIndexInterval(cf_def.min_index_interval); + if (cf_def.isSetMax_index_interval()) + newCFMD.maxIndexInterval(cf_def.max_index_interval); + if (cf_def.isSetSpeculative_retry()) + newCFMD.speculativeRetry(CFMetaData.SpeculativeRetry.fromString(cf_def.speculative_retry)); + if (cf_def.isSetTriggers()) + newCFMD.triggers(triggerDefinitionsFromThrift(cf_def.triggers)); + + return newCFMD.comment(cf_def.comment) + .defaultValidator(TypeParser.parse(cf_def.default_validation_class)) + .compressionParameters(CompressionParameters.create(cf_def.compression_options)) + .rebuild(); + } + catch (SyntaxException | MarshalException e) + { + throw new ConfigurationException(e.getMessage()); + } + } + + /** applies implicit defaults to cf definition. useful in updates */ + private static void applyImplicitDefaults(org.apache.cassandra.thrift.CfDef cf_def) + { + if (!cf_def.isSetComment()) + cf_def.setComment(""); + if (!cf_def.isSetMin_compaction_threshold()) + cf_def.setMin_compaction_threshold(CFMetaData.DEFAULT_MIN_COMPACTION_THRESHOLD); + if (!cf_def.isSetMax_compaction_threshold()) + cf_def.setMax_compaction_threshold(CFMetaData.DEFAULT_MAX_COMPACTION_THRESHOLD); + if (cf_def.compaction_strategy == null) + cf_def.compaction_strategy = CFMetaData.DEFAULT_COMPACTION_STRATEGY_CLASS.getSimpleName(); + if (cf_def.compaction_strategy_options == null) + cf_def.compaction_strategy_options = Collections.emptyMap(); + if (!cf_def.isSetCompression_options()) + cf_def.setCompression_options(Collections.singletonMap(CompressionParameters.SSTABLE_COMPRESSION, CFMetaData.DEFAULT_COMPRESSOR)); + if (!cf_def.isSetDefault_time_to_live()) + cf_def.setDefault_time_to_live(CFMetaData.DEFAULT_DEFAULT_TIME_TO_LIVE); + if (!cf_def.isSetDclocal_read_repair_chance()) + cf_def.setDclocal_read_repair_chance(CFMetaData.DEFAULT_DCLOCAL_READ_REPAIR_CHANCE); + + // if index_interval was set, use that for the min_index_interval default + if (!cf_def.isSetMin_index_interval()) + { + if (cf_def.isSetIndex_interval()) + cf_def.setMin_index_interval(cf_def.getIndex_interval()); + else + cf_def.setMin_index_interval(CFMetaData.DEFAULT_MIN_INDEX_INTERVAL); + } + + if (!cf_def.isSetMax_index_interval()) + { + // ensure the max is at least as large as the min + cf_def.setMax_index_interval(Math.max(cf_def.min_index_interval, CFMetaData.DEFAULT_MAX_INDEX_INTERVAL)); + } + } + + /** + * Create CFMetaData from thrift {@link CqlRow} that contains columns from schema_columnfamilies. + * + * @param columnsRes CqlRow containing columns from schema_columnfamilies. + * @return CFMetaData derived from CqlRow + */ + public static CFMetaData fromThriftCqlRow(CqlRow cf, CqlResult columnsRes) + { + UntypedResultSet.Row cfRow = new UntypedResultSet.Row(convertThriftCqlRow(cf)); + + List> cols = new ArrayList<>(columnsRes.rows.size()); + for (CqlRow row : columnsRes.rows) + cols.add(convertThriftCqlRow(row)); + UntypedResultSet colsRows = UntypedResultSet.create(cols); + + return LegacySchemaTables.createTableFromTableRowAndColumnRows(cfRow, colsRows); + } + + private static Map convertThriftCqlRow(CqlRow row) + { + Map m = new HashMap<>(); + for (org.apache.cassandra.thrift.Column column : row.getColumns()) + m.put(UTF8Type.instance.getString(column.bufferForName()), column.value); + return m; + } + + public static CfDef toThrift(CFMetaData cfm) + { + CfDef def = new CfDef(cfm.ksName, cfm.cfName); + def.setColumn_type(cfm.cfType.name()); + + if (cfm.isSuper()) + { + def.setComparator_type(cfm.comparator.subtype(0).toString()); + def.setSubcomparator_type(cfm.comparator.subtype(1).toString()); + } + else + { + def.setComparator_type(cfm.comparator.toString()); + } + + def.setComment(Strings.nullToEmpty(cfm.getComment())); + def.setRead_repair_chance(cfm.getReadRepairChance()); + def.setDclocal_read_repair_chance(cfm.getDcLocalReadRepairChance()); + def.setGc_grace_seconds(cfm.getGcGraceSeconds()); + def.setDefault_validation_class(cfm.getDefaultValidator().toString()); + def.setKey_validation_class(cfm.getKeyValidator().toString()); + def.setMin_compaction_threshold(cfm.getMinCompactionThreshold()); + def.setMax_compaction_threshold(cfm.getMaxCompactionThreshold()); + // We only return the alias if only one is set since thrift don't know about multiple key aliases + if (cfm.partitionKeyColumns().size() == 1) + def.setKey_alias(cfm.partitionKeyColumns().get(0).name.bytes); + def.setColumn_metadata(columnDefinitionsToThrift(cfm.allColumns())); + def.setCompaction_strategy(cfm.compactionStrategyClass.getName()); + def.setCompaction_strategy_options(new HashMap<>(cfm.compactionStrategyOptions)); + def.setCompression_options(cfm.compressionParameters.asThriftOptions()); + def.setBloom_filter_fp_chance(cfm.getBloomFilterFpChance()); + def.setMin_index_interval(cfm.getMinIndexInterval()); + def.setMax_index_interval(cfm.getMaxIndexInterval()); + def.setMemtable_flush_period_in_ms(cfm.getMemtableFlushPeriod()); + def.setCaching(cfm.getCaching().toThriftCaching()); + def.setCells_per_row_to_cache(cfm.getCaching().toThriftCellsPerRow()); + def.setDefault_time_to_live(cfm.getDefaultTimeToLive()); + def.setSpeculative_retry(cfm.getSpeculativeRetry().toString()); + def.setTriggers(triggerDefinitionsToThrift(cfm.getTriggers().values())); + + return def; + } + + public static ColumnDefinition fromThrift(String ksName, + String cfName, + AbstractType thriftComparator, + AbstractType thriftSubcomparator, + ColumnDef thriftColumnDef) + throws SyntaxException, ConfigurationException + { + // For super columns, the componentIndex is 1 because the ColumnDefinition applies to the column component. + Integer componentIndex = thriftSubcomparator != null ? 1 : null; + AbstractType comparator = thriftSubcomparator == null ? thriftComparator : thriftSubcomparator; + try + { + comparator.validate(thriftColumnDef.name); + } + catch (MarshalException e) + { + throw new ConfigurationException(String.format("Column name %s is not valid for comparator %s", ByteBufferUtil.bytesToHex(thriftColumnDef.name), comparator)); + } + + return new ColumnDefinition(ksName, + cfName, + new ColumnIdentifier(ByteBufferUtil.clone(thriftColumnDef.name), comparator), + TypeParser.parse(thriftColumnDef.validation_class), + thriftColumnDef.index_type == null ? null : org.apache.cassandra.config.IndexType.valueOf(thriftColumnDef.index_type.name()), + thriftColumnDef.index_options, + thriftColumnDef.index_name, + componentIndex, + ColumnDefinition.Kind.REGULAR); + } + + private static List fromThrift(String ksName, + String cfName, + AbstractType thriftComparator, + AbstractType thriftSubcomparator, + List thriftDefs) + throws SyntaxException, ConfigurationException + { + if (thriftDefs == null) - return Collections.emptyList(); ++ return new ArrayList<>(); + + List defs = new ArrayList<>(thriftDefs.size()); + for (ColumnDef thriftColumnDef : thriftDefs) + defs.add(fromThrift(ksName, cfName, thriftComparator, thriftSubcomparator, thriftColumnDef)); + + return defs; + } + + @VisibleForTesting + public static ColumnDef toThrift(ColumnDefinition column) + { + ColumnDef cd = new ColumnDef(); + + cd.setName(ByteBufferUtil.clone(column.name.bytes)); + cd.setValidation_class(column.type.toString()); + cd.setIndex_type(column.getIndexType() == null ? null : org.apache.cassandra.thrift.IndexType.valueOf(column.getIndexType().name())); + cd.setIndex_name(column.getIndexName()); + cd.setIndex_options(column.getIndexOptions() == null ? null : Maps.newHashMap(column.getIndexOptions())); + + return cd; + } + + private static List columnDefinitionsToThrift(Collection columns) + { + List thriftDefs = new ArrayList<>(columns.size()); + for (ColumnDefinition def : columns) + if (def.kind == ColumnDefinition.Kind.REGULAR) + thriftDefs.add(ThriftConversion.toThrift(def)); + return thriftDefs; + } + + private static Map triggerDefinitionsFromThrift(List thriftDefs) + { + Map triggerDefinitions = new HashMap<>(); + for (TriggerDef thriftDef : thriftDefs) + triggerDefinitions.put(thriftDef.getName(), + new TriggerDefinition(thriftDef.getName(), thriftDef.getOptions().get(TriggerDefinition.CLASS))); + return triggerDefinitions; + } + + private static List triggerDefinitionsToThrift(Collection triggers) + { + List thriftDefs = new ArrayList<>(triggers.size()); + for (TriggerDefinition def : triggers) + { + TriggerDef td = new TriggerDef(); + td.setName(def.name); + td.setOptions(Collections.singletonMap(TriggerDefinition.CLASS, def.classOption)); + thriftDefs.add(td); + } + return thriftDefs; + } }