cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Castelain, Alain" <Alain.Castel...@xerox.com>
Subject Why a cluster don't start after cassandra.yaml range_timeout parameter change ?
Date Wed, 03 Dec 2014 14:57:07 GMT


Hi,

I had a three node cluster in cassandra 1.2 16 version running well
So I have changed the range_request_timeout_in_ms form 10000 to 20000 on two nodes and this
nodes restarted well.
On the last node I received this messages from the output.log file :

 INFO 19:27:18,691 Cassandra shutting down...
 INFO 19:27:18,706 Stop listening to thrift clients
 INFO 19:27:18,905 Stop listening for CQL clients
 INFO 19:27:25,153 Announcing shutdown
 INFO 19:27:25,244 GC for ConcurrentMarkSweep: 6227 ms for 3 collections, 1200395040 used;
max is 21045379072
 INFO 19:27:25,244 Pool Name                    Active   Pending      Completed   Blocked
 All Time Blocked
 INFO 19:27:25,245 ReadStage                         0         0        7960169         0
                0
 INFO 19:27:25,245 RequestResponseStage              0         0       29973176         0
                0
 INFO 19:27:25,245 ReadRepairStage                   0         0        4741703         0
                0
 INFO 19:27:25,245 MutationStage                     0         0       26456216         0
                0
 INFO 19:27:25,245 ReplicateOnWriteStage             0         0             13         0
                0
 INFO 19:27:25,245 GossipStage                       0         0        1656431         0
                0
 INFO 19:27:25,245 AntiEntropyStage                  0         0          84461         0
                0
 INFO 19:27:25,246 MigrationStage                    0         0        2534350         0
                0
 INFO 19:27:25,246 MemtablePostFlusher               0         0          40644         0
                0
 INFO 19:27:25,246 MemoryMeter                       0         0            145         0
                0
 INFO 19:27:25,246 FlushWriter                       0         0           2869         0
               20
 INFO 19:27:25,246 MiscStage                         0         0           1651         0
                0
 INFO 19:27:25,246 PendingRangeCalculator            0         0              6         0
                0
 INFO 19:27:25,246 commitlog_archiver                0         0              0         0
                0
 INFO 19:27:25,246 InternalResponseStage             0         0        4633968         0
                0
 INFO 19:27:25,247 AntiEntropySessions               0         0          15360         0
                0
 INFO 19:27:25,247 HintedHandoff                     0         0            200         0
                0
 INFO 19:27:25,247 CompactionManager                 0         0
 INFO 19:27:25,247 Commitlog                       n/a         0
 INFO 19:27:25,247 MessagingService                n/a       0/0
 INFO 19:27:25,247 Cache Type                     Size                 Capacity          
    KeysToSave                                                         Provider
 INFO 19:27:25,247 KeyCache                  104857412                104857600          
           all
 INFO 19:27:25,247 RowCache                          0                        0          
           all              org.apache.cassandra.cache.SerializingCacheProvider
 INFO 19:27:25,247 ColumnFamily                Memtable ops,data
 INFO 19:27:25,248 Test_Sib.Files                            0,0
 INFO 19:27:25,248 idfx.report_file_counters                 0,0
 INFO 19:27:25,248 idfx.report_file                          0,0
 INFO 19:27:25,248 SibBackup2.Files                          0,0
 INFO 19:27:25,248 SibTss2.Files                             0,0
 INFO 19:27:25,248 SibLeroyMerlin.Files            6316,87364119
 INFO 19:27:25,248 SibCasto.Files                            0,0
 INFO 19:27:25,248 idfxr01.system_properties                 0,0
 INFO 19:27:25,248 idfxr01.edgestore                         0,0
 INFO 19:27:25,248 idfxr01.edgeindex                         0,0
 INFO 19:27:25,248 idfxr01.titan_ids                         0,0
 INFO 19:27:25,248 idfxr01.edgestore_lock_                   0,0
 INFO 19:27:25,248 idfxr01.vertexindex                       0,0
 INFO 19:27:25,248 idfxr01.vertexindex_lock_                 0,0
 INFO 19:27:25,248 SibBackupTest.Files                       0,0
 INFO 19:27:25,248 SibNorauto.SibCassandraColumnFamilyName      1444,753849990
 INFO 19:27:25,248 SibNorauto.Files                          0,0
 INFO 19:27:25,248 SibTss.Files                              0,0
 INFO 19:27:25,248 SibTss1.Files                             0,0
 INFO 19:27:25,249 OpsCenter.bestpractice_results                 0,0
 INFO 19:27:25,249 OpsCenter.events                      12,3520
 INFO 19:27:25,249 OpsCenter.rollups60           406517,77862510
 INFO 19:27:25,249 OpsCenter.settings                  216,38195
 INFO 19:27:25,249 OpsCenter.pdps                168056,12582912
 INFO 19:27:25,249 OpsCenter.rollups86400                    0,0
 INFO 19:27:25,249 OpsCenter.events_timeline               2,320
 INFO 19:27:25,249 OpsCenter.rollups300           81242,44035928
 INFO 19:27:25,249 OpsCenter.rollups7200            4365,1488742
 INFO 19:27:25,249 SibLamy.Files                 24750,800384035
 INFO 19:27:25,249 SibLamy.SibCassandraColumnFamilyName                 0,0
 INFO 19:27:25,249 system.local                            2,520
 INFO 19:27:25,249 system.peers                          16,1860
 INFO 19:27:25,249 system.batchlog                           0,0
 INFO 19:27:25,249 system.NodeIdInfo                         0,0
 INFO 19:27:25,249 system.LocationInfo                       0,0
 INFO 19:27:25,249 system.Schema                             0,0
 INFO 19:27:25,249 system.Migrations                         0,0
 INFO 19:27:25,249 system.schema_keyspaces                   0,0
 INFO 19:27:25,250 system.schema_columns                     0,0
 INFO 19:27:25,250 system.schema_columnfamilies                 0,0
 INFO 19:27:25,250 system.IndexInfo                          0,0
 INFO 19:27:25,250 system.range_xfers                        0,0
 INFO 19:27:25,250 system.peer_events                        0,0
 INFO 19:27:25,250 system.hints                      299,2880396
 INFO 19:27:25,250 system.HintsColumnFamily                  0,0
 INFO 19:27:25,250 SibBackup.Files                59564,53786449
 INFO 19:27:25,250 SibBytel.Files                            0,0
 INFO 19:27:25,250 SibColas.Files                            0,0
 INFO 19:27:25,250 SibCassandraPierre.Files                  0,0
 INFO 19:27:25,250 SibVeolia.Files                           0,0
 INFO 19:27:25,250 idfxp01.vertexindex                       0,0
 INFO 19:27:25,250 idfxp01.vertexindex_lock_                 0,0
 INFO 19:27:25,250 idfxp01.edgestore                         0,0
 INFO 19:27:25,250 idfxp01.edgeindex                         0,0
 INFO 19:27:25,250 idfxp01.system_properties                 0,0
 INFO 19:27:25,250 idfxp01.edgestore_lock_                   0,0
 INFO 19:27:25,250 idfxp01.titan_ids                         0,0
 INFO 19:27:25,251 system_traces.sessions                    0,0
 INFO 19:27:25,251 system_traces.events                      0,0
 INFO 19:27:25,251 SibCassandraTestPierre.Files                 0,0
 INFO 19:27:27,153 Waiting for messaging service to quiesce
 INFO 19:27:27,159 MessagingService shutting down server thread.

After the service cassandra start command :

 INFO 19:27:33,739 Logging initialized
 INFO 19:27:33,768 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_55
 INFO 19:27:33,770 Heap size: 21045379072/21045379072
 INFO 19:27:33,770 Classpath: /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.6.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-13.0.1.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.1.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.2.16.jar:/usr/share/cassandra/apache-cassandra-thrift-1.2.16.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
 INFO 19:27:41,717 JNA mlockall successful
 INFO 19:27:41,739 Loading settings from file:/etc/cassandra/cassandra.yaml
 INFO 19:27:42,052 Data files directories: [/data/cassandra/data]
 INFO 19:27:42,052 Commit log directory: /commitlog
 INFO 19:27:42,052 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
 INFO 19:27:42,052 disk_failure_policy is stop
 INFO 19:27:42,058 Global memtable threshold is enabled at 6690MB
 INFO 19:27:42,263 Not using multi-threaded compaction
 INFO 19:27:42,799 Initializing key cache with capacity of 100 MBs.
 INFO 19:27:42,818 Scheduling key cache save to each 14400 seconds (going to save all keys).
 INFO 19:27:42,820 Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider
 INFO 19:27:42,820 Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider
 INFO 19:27:42,831 Scheduling row cache save to each 0 seconds (going to save all keys).
 INFO 19:27:43,269 Initializing system.batchlog
 INFO 19:27:43,318 Initializing system.peer_events
 INFO 19:27:43,327 Initializing system.hints
 INFO 19:27:43,336 Opening /data/cassandra/data/system/hints/system-hints-ic-22 (7813966 bytes)
 INFO 19:27:43,337 Opening /data/cassandra/data/system/hints/system-hints-ic-22 (7813966 bytes)
 INFO 19:27:43,436 Initializing system.Schema
 INFO 19:27:43,445 Initializing system.schema_keyspaces
 INFO 19:27:43,449 Opening /data/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-208
(623 bytes)
 INFO 19:27:43,449 Opening /data/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-207
(703 bytes)
 INFO 19:27:43,450 Opening /data/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-208
(623 bytes)
 INFO 19:27:43,450 Opening /data/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-207
(703 bytes)
 INFO 19:27:43,556 reading saved cache /data/cassandra/saved_caches/system-schema_keyspaces-KeyCache-b.db
 INFO 19:27:43,568 Initializing system.range_xfers
 INFO 19:27:43,574 Initializing system.HintsColumnFamily
 INFO 19:27:43,583 Initializing system.schema_columnfamilies
 INFO 19:27:43,587 Opening /data/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-265
(15533 bytes)
 INFO 19:27:43,587 Opening /data/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-266
(10706 bytes)
 INFO 19:27:43,587 Opening /data/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-265
(15533 bytes)
 INFO 19:27:43,587 Opening /data/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-266
(10706 bytes)
 INFO 19:27:43,640 reading saved cache /data/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db
 INFO 19:27:43,650 Initializing system.NodeIdInfo
 INFO 19:27:43,652 Opening /data/cassandra/data/system/NodeIdInfo/system-NodeIdInfo-ic-1 (64
bytes)
 INFO 19:27:43,652 Opening /data/cassandra/data/system/NodeIdInfo/system-NodeIdInfo-ic-1 (64
bytes)
 INFO 19:27:43,680 Initializing system.schema_columns
 INFO 19:27:43,682 Opening /data/cassandra/data/system/schema_columns/system-schema_columns-ic-113
(3266 bytes)
 INFO 19:27:43,683 Opening /data/cassandra/data/system/schema_columns/system-schema_columns-ic-113
(3266 bytes)
 INFO 19:27:43,689 reading saved cache /data/cassandra/saved_caches/system-schema_columns-KeyCache-b.db
 INFO 19:27:43,704 Initializing system.IndexInfo
 INFO 19:27:43,708 Opening /data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-6 (156
bytes)
 INFO 19:27:43,709 Opening /data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-7 (72
bytes)
 INFO 19:27:43,709 Opening /data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-5 (71
bytes)
 INFO 19:27:43,709 Opening /data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-6 (156
bytes)
 INFO 19:27:43,710 Opening /data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-5 (71
bytes)
 INFO 19:27:43,709 Opening /data/cassandra/data/system/IndexInfo/system-IndexInfo-ic-7 (72
bytes)
 INFO 19:27:43,740 Initializing system.Migrations
 INFO 19:27:43,748 Initializing system.peers
 INFO 19:27:43,751 Opening /data/cassandra/data/system/peers/system-peers-ic-131 (213 bytes)
 INFO 19:27:43,752 Opening /data/cassandra/data/system/peers/system-peers-ic-130 (19380 bytes)
 INFO 19:27:43,752 Opening /data/cassandra/data/system/peers/system-peers-ic-131 (213 bytes)
 INFO 19:27:43,752 Opening /data/cassandra/data/system/peers/system-peers-ic-130 (19380 bytes)
 INFO 19:27:43,846 Initializing system.local
 INFO 19:27:43,849 Opening /data/cassandra/data/system/local/system-local-ic-265 (84 bytes)
 INFO 19:27:43,849 Opening /data/cassandra/data/system/local/system-local-ic-264 (10379 bytes)
 INFO 19:27:43,850 Opening /data/cassandra/data/system/local/system-local-ic-265 (84 bytes)
 INFO 19:27:43,850 Opening /data/cassandra/data/system/local/system-local-ic-264 (10379 bytes)
 INFO 19:27:43,869 reading saved cache /data/cassandra/saved_caches/system-local-KeyCache-b.db
 INFO 19:27:43,895 Initializing system.LocationInfo
java.lang.AssertionError
        at org.apache.cassandra.cql3.CFDefinition.<init>(CFDefinition.java:162)
        at org.apache.cassandra.config.CFMetaData.updateCfDef(CFMetaData.java:1541)
        at org.apache.cassandra.config.CFMetaData.fromSchema(CFMetaData.java:1456)
        at org.apache.cassandra.config.KSMetaData.deserializeColumnFamilies(KSMetaData.java:306)
        at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:287)
        at org.apache.cassandra.db.DefsTable.loadFromTable(DefsTable.java:154)
        at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:588)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:253)
        at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:381)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606) 
	at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212) 
Cannot load daemon 
Service exit with a return value of 3

After I tried to clean commit log and saved_cahes directories and restart with no effect

I have cleaned the all  data directory and restarted the node with auto_bootstrap and the
node retrieved only two keyspaces .
A this point the previous error comes back when I try to restart without auto_bootstrap parameter.

If you could help me I don't have any idea for this node.

The other nodes are running properly.

Many thank's for your help.   



Cordialement, Regards,

Alain Castelain
Database Administrator
Xerox - Global Document Outsourcing
253, avenue du Président Wilson
La Plaine Saint Denis Cedex, 93211
France
http://www.xerox.fr/services/frfr.html

p (+33) 155 847 333
m (+33) 682 999 617
@ alain.castelain@xerox.com




Mime
View raw message