cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From CassUser CassUser <cassu...@gmail.com>
Subject Re: Starting up cassandra 0.7
Date Wed, 17 Nov 2010 20:10:56 GMT
We are putting together a patch for that.  Also updating the unit test to
capture this case.  It wasn't failing as a result of it being backwards.

On Wed, Nov 17, 2010 at 11:51 AM, CassUser CassUser <cassuser@gmail.com>wrote:

> Thanks for looking into that. I created a jira issue for it:
>
> https://issues.apache.org/jira/browse/CASSANDRA-1753
>
>
>
> On Wed, Nov 17, 2010 at 11:28 AM, Aaron Morton <aaron@thelastpickle.com>wrote:
>
>> That does look like a bug
>> http://www.mail-archive.com/dev@cassandra.apache.org/msg01167.html
>>
>>  <http://www.mail-archive.com/dev@cassandra.apache.org/msg01167.html>
>> Aaron
>>
>>
>> On 17 Nov, 2010,at 03:55 PM, Aaron Morton <aaron@thelastpickle.com>
>> wrote:
>>
>> It's certainly looks suspect. I've had a look at the code around
>> SSTableImport and SSTableExport and the isDeleted value for the col is based
>> on IColumn.isMarkedForDelete read when the data was exported. I'll try to
>> have a look tonight, or if someone is still up in the states they may help.
>> The current 0.7 branch has the same code.
>>
>> Are there any other errors in the log?
>>
>> The code sample you've got there uses the TimestampClocks that were once a
>> part of the 0.7 development, but have been removed. So your data sample may
>> not be usable going forwards. Can you jump to beta3?
>>
>> Aaron
>>
>>
>>
>> On 17 Nov, 2010,at 02:03 PM, CassUser CassUser <cassuser@gmail.com>
>> wrote:
>>
>> Looking at this closer.  I noticed the following in the SSTableImport
>> Class:
>>
>>             if (col.isDeleted) {
>>                 cfamily.addColumn(path, hexToBytes(col.value), new
>> TimestampClock(col.timestamp));
>>             } else {
>>                 cfamily.addTombstone(path, hexToBytes(col.value), new
>> TimestampClock(col.timestamp));
>>             }
>>
>> This appears to be backwards.
>>
>>
>> On Tue, Nov 16, 2010 at 4:03 PM, CassUser CassUser <cassuser@gmail.com>wrote:
>>
>>> Looked at how DatabaseDescriptor is loading the yaml file.  Using that
>>> approach solves the problem with the column_families mapping exception.
>>>
>>> The problems we are running into currently is regarding a known dataset
>>> not being loaded into our test instance correctly.
>>>
>>> Steps:
>>> 1.  Create temp directory to host cassandra test instance
>>> 2.  Create keyspace directories
>>> 3  Update yaml file, and copy to test location
>>> 4.  Add schema from yaml using DatabaseDescriptor.
>>> readTablesFromYaml() & DatabaseDescriptor.
>>> setTableDefinition
>>> 4.  Use SSTableImport.importJson to setup a known dataset.  We have JSON
>>> files converted to 0.7 compliant (byte [] rows)
>>> 5.  Start Cassandra instance using the EmbeddedCassandraService class
>>>
>>> Everything appears to work from the log messages.  I get the message "
>>> Sampling index for ..", and finally "Listening for thrift clients...".  When
>>> i use a client to query the data in test instance, I notice keys with no
>>> columns/values stored (via keyrange slice).  Using a open ended column slice
>>> for a known key I'm getting zero results.
>>>
>>> I'm currently running cassandra 0.7. beta2.  Is the steps I've outlined
>>> above suppose to work in 0.7?  We did something similar in 0.6.4 without any
>>> problems.  Is there known bugs I can look into?
>>>
>>> Thanks.
>>>
>>>
>>>
>>>
>>> On Tue, Nov 16, 2010 at 12:24 PM, Aaron Morton <aaron@thelastpickle.com>wrote:
>>>
>>>> I've not used the embedded service.
>>>>
>>>> The code in o.a.c.service.EmbeddedCassandraService says it will read the
>>>> yaml file. If the cluster does not have a schema stored I think it will load
>>>> the one from yaml.
>>>>
>>>> Have you tried starting it up with an empty system data dir ? Does it
>>>> pickup the schema from the yaml?
>>>>
>>>> Aaron
>>>>
>>>>
>>>> On 17 Nov, 2010,at 09:17 AM, CassUser CassUser <cassuser@gmail.com>
>>>> wrote:
>>>>
>>>>
>>>> Loading yaml file like so:
>>>>         FileInputStream yamlInputStream = new FileInputStream(
>>>>                 configTemplateFile);
>>>>         Constructor constructor = new Constructor(Config.class);
>>>>         Yaml yaml = new Yaml(new Loader(constructor));
>>>>         Config conf = (Config) yaml.load(yamlInputStream);
>>>>
>>>> Fails on the last line.
>>>>
>>>> Although if i have the CFs defined like this:
>>>> keyspaces:
>>>> - column_families:
>>>>   - !!org.apache.cassandra.config.RawColumnFamily
>>>>     column_metadata: []
>>>>     column_type: null
>>>>     comment: null
>>>>     compare_subcolumns_with: null
>>>>     compare_with: BytesType
>>>>     default_validation_class: null
>>>>     gc_grace_seconds: 864000
>>>>     keys_cached: 200000.0
>>>>     max_compaction_threshold: 32
>>>>     min_compaction_threshold: 4
>>>>     name: Similarity
>>>>     preload_row_cache: false
>>>>     read_repair_chance: 1.0
>>>>     rows_cached: 0.0
>>>>
>>>>
>>>> it appears to startup.  Any idea whats going on here?
>>>>
>>>>
>>>>
>>>> On Tue, Nov 16, 2010 at 11:58 AM, CassUser CassUser <cassuser@gmail.com
>>>> > wrote:
>>>>
>>>>>
>>>>> This is embedded for testing cassandra 0.7 beta2.  using
>>>>> EmbeddedCassandraService.
>>>>>
>>>>> and manually adding schema programmatically using:
>>>>>         for (KSMetaData table :
>>>>> DatabaseDescriptor.readTablesFromYaml()) {
>>>>>             for (CFMetaData cfm : table.cfMetaData().values()) {
>>>>>                 CFMetaData.map(cfm);
>>>>>             }
>>>>>             DatabaseDescriptor.setTableDefinition(table,
>>>>>                     DatabaseDescriptorgetDefsVersion());
>>>>>         }
>>>>>
>>>>> Is this the correct way to start up a test server, with the schema
>>>>> loaded?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Nov 16, 2010 at 11:41 AM, Aaron Morton <
>>>>> aaron@thelastpickle.com> wrote:
>>>>>
>>>>>>
>>>>>> AFAIK the ArrayStoreException is similar to a type mismatch Is it
>>>>>> possible you have something mixed up in your class path or source
code if
>>>>>> you built from source?
>>>>>>
>>>>>> It looks like the column family info was deserialised into a
>>>>>> o.a.c.config.RawColumnFamily but when that object was added to the
>>>>>> RawColumnFamily[] array on o.a.c.config.RawKeyspace if was the wrong
type.
>>>>>>
>>>>>> Have you tried a clean build ? There are some things in the call
stack
>>>>>> which look custom, are you starting from the command line or is this
>>>>>> embedded for testing?
>>>>>>
>>>>>> Hope that helps.
>>>>>> Aaron
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 17 Nov, 2010,at 08:09 AM, CassUser CassUser <cassuser@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Here is the yaml:
>>>>>>
>>>>>> # Cassandra YAML generated from previous config
>>>>>> # Configuration wiki:
>>>>>> http://wiki.apache.org/cassandra/StorageConfiguration
>>>>>>  authenticator: org.apache.cassandraauth.AllowAllAuthenticator
>>>>>>
>>>>>>
>>>>>> auto_bootstrap: false
>>>>>> binary_memtable_throughput_in_mb: 256
>>>>>> cluster_name: Test Cluster
>>>>>> column_index_size_in_kb: 64
>>>>>> commitlog_rotation_threshold_in_mb: 128
>>>>>> commitlog_sync: periodic
>>>>>> commitlog_sync_period_in_ms: 10000
>>>>>> compaction_thread_priority: 1
>>>>>> concurrent_reads: 8
>>>>>> concurrent_writes: 32
>>>>>> disk_access_mode: auto
>>>>>> dynamic_snitch: false
>>>>>> endpoint_snitch: org.apache.cassandra.locator.SimpleSnitch
>>>>>> request_scheduler: org.apache.cassandra.scheduler.RoundRobinScheduler
>>>>>> request_scheduler_id: keyspace
>>>>>> hinted_handoff_enabled: true
>>>>>> in_memory_compaction_limit_in_mb: 128
>>>>>> index_interval: 128
>>>>>> keyspaces:
>>>>>>     - name: myKeyspace
>>>>>>
>>>>>>       replica_placement_strategy:
>>>>>> orgapache.cassandra.locator.SimpleStrategy
>>>>>>
>>>>>>
>>>>>>       replication_factor: 1
>>>>>>       column_families:
>>>>>>         - name: Standard1
>>>>>>           rows_cached: 100
>>>>>>           keys_cached: 1
>>>>>>           compare_with: UTF8Type
>>>>>> memtable_flush_after_mins: 60
>>>>>> memtable_operations_in_millions: 0.3
>>>>>> memtable_throughput_in_mb: 64
>>>>>> partitioner: org.apache.cassandra.dht.RandomPartitioner
>>>>>> phi_convict_threshold: 8
>>>>>> rpc_keepalive: true
>>>>>> rpc_port: 9160
>>>>>> rpc_timeout_in_ms: 10000
>>>>>> sliced_buffer_size_in_kb: 64
>>>>>> snapshot_before_compaction: false
>>>>>> storage_port: 7000
>>>>>> thrift_framed_transport_size_in_mb: 15
>>>>>> thrift_max_message_length_in_mb: 16
>>>>>>
>>>>>> and here is the exception i'm receiving:
>>>>>>
>>>>>> Caused by: Can't construct a java object for tag:yaml.org,2002:org.apachecassandra.config.Config;
>>>>>> exception=Cannot create property=keyspaces for Java
>>>>>> Bean=org.apache.cassandra.config.Config@12e43f1; Cannot create
>>>>>> property=column_families for
>>>>>> JavaBean=org.apache.cassandra.config.RawKeyspace@1a8bd74;
>>>>>> null
>>>>>>  in "<reader>", line 3, column 1:
>>>>>>     authenticator: org.apache.cassan ...
>>>>>>     ^
>>>>>>
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:372)
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructorjava:177)
>>>>>>          at
>>>>>> org.yaml.snakeyamlconstructor.BaseConstructorconstructDocument(BaseConstructor.java:136)
>>>>>>
>>>>>>
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:122)
>>>>>>         at org.yaml.snakeyaml.Loader.load(Loader.java:52)
>>>>>>         at org.yaml.snakeyaml.Yaml.load(Yaml.java:166)
>>>>>>         at
>>>>>> com.atsid.cassandra.testutils.CassandraTestRunner.updateYamlConfig(CassandraTestRunner.java:134)
>>>>>>         at
>>>>>> com.atsid.cassandra.testutils.CassandraTestRunner.init(CassandraTestRunner.java:119)
>>>>>>         at
>>>>>> com.atsid.cassandra.testutils.CassandraTestRunner.main(CassandraTestRunner.java:272)
>>>>>>         ... 6 more
>>>>>> Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create
>>>>>> property=keyspaces for JavaBean=org.apache.cassandraconfig.Config@12e43f1;
>>>>>> Cannot cr
>>>>>> eate property=column_families for
>>>>>> JavaBean=org.apache.cassandra.config.RawKeyspace@1a8bd74; null
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:305)
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:184)
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:370)
>>>>>>         ... 14 more
>>>>>> Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create
>>>>>> property=column_families for
>>>>>> JavaBean=org.apache.cassandra.config.RawKeyspace@1a8bd74
>>>>>> ; null
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:305)
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:184)
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:177)
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructor.BaseConstructor.constructSequenceStep2(BaseConstructor.java:239)
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructor.BaseConstructor.constructSequence(BaseConstructor.java:233)
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructorConstructor$ConstructSequence.construct(Constructor.java:543)
>>>>>>
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructor.BaseConstructorconstructObject(BaseConstructor.java:177)
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:298)
>>>>>>         ... 16 more
>>>>>> Caused by: java.lang.ArrayStoreException
>>>>>>         at java.lang.System.arraycopy(Native Method)
>>>>>>         at java.util.Arrays.copyOf(Arrays.java:2763)
>>>>>>         at java.util.ArrayList.toArray(ArrayList.java:305)
>>>>>>         at
>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:301)
>>>>>>         ... 23 more
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message