cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cogumelos Maravilha <cogumelosmaravi...@sapo.pt>
Subject Re: Node always dieing
Date Mon, 10 Apr 2017 11:48:23 GMT
No.

nodetool status, nodetool describecluster also nodetool ring shows a
correct cluster.

Not all nodes needs to be a seed, but can be.

I had also ALTER KEYSPACE system_auth WITH REPLICATION = { 'class' :
'SimpleStrategy', 'replication_factor' : 6 } AND durable_writes = false;

And the first command on the new node was  |nodetool repair system_auth|


On 04/10/2017 12:37 PM, Chris Mawata wrote:
> Notice
> .SimpleSeedProvider{seeds=10.100.100.19, 10.100.100.85,
> 10.100.100.185, 10.100.100.161, 10.100.100.52, 10.100.1000.213};
>
> Why do you have all six of your nodes as seeds? is it possible that
> the last one you added used itself as the seed and is isolated?
>
> On Thu, Apr 6, 2017 at 6:48 AM, Cogumelos Maravilha
> <cogumelosmaravilha@sapo.pt <mailto:cogumelosmaravilha@sapo.pt>> wrote:
>
>     Yes C* is running as cassandra:
>
>     cassand+  2267     1 99 10:18 ?        00:02:56 java
>     -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities
>     -XX:Threa...
>
>     INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
>     configuration:[allocate_tokens_for_keyspace=null;
>     authenticator=PasswordAuthenticator;
>     authorizer=CassandraAuthorizer; auto_bootstrap=true;
>     auto_snapshot=true; back_pressure_enabled=false;
>     back_pressure_strategy=org.apache.cassandra.net
>     <http://apache.cassandra.net>.RateBasedBackPressure{high_ratio=0.9,
>     factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
>     batch_size_warn_threshold_in_kb=5;
>     batchlog_replay_throttle_in_kb=1024; broadcast_address=null;
>     broadcast_rpc_address=null;
>     buffer_pool_use_heap_if_exhausted=true;
>     cas_contention_timeout_in_ms=6000000; cdc_enabled=false;
>     cdc_free_space_check_interval_ms=250; cdc_raw_directory=null;
>     cdc_total_space_in_mb=0; client_encryption_options=<REDACTED>;
>     cluster_name=company; column_index_cache_size_in_kb=2;
>     column_index_size_in_kb=64; commit_failure_policy=ignore;
>     commitlog_compression=null;
>     commitlog_directory=/mnt/cassandra/commitlog;
>     commitlog_max_compression_buffers_in_pool=3;
>     commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
>     commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
>     commitlog_sync_period_in_ms=10000;
>     commitlog_total_space_in_mb=null;
>     compaction_large_partition_warning_threshold_mb=100;
>     compaction_throughput_mb_per_sec=16; concurrent_compactors=null;
>     concurrent_counter_writes=32;
>     concurrent_materialized_view_writes=32; concurrent_reads=32;
>     concurrent_replicates=null; concurrent_writes=32;
>     counter_cache_keys_to_save=2147483647;
>     counter_cache_save_period=7200; counter_cache_size_in_mb=null;
>     counter_write_request_timeout_in_ms=6000000;
>     credentials_cache_max_entries=1000;
>     credentials_update_interval_in_ms=-1;
>     credentials_validity_in_ms=2000; cross_node_timeout=false;
>     data_file_directories=[Ljava.lang.String;@223f3642;
>     disk_access_mode=auto; disk_failure_policy=ignore;
>     disk_optimization_estimate_percentile=0.95;
>     disk_optimization_page_cross_chance=0.1;
>     disk_optimization_strategy=ssd; dynamic_snitch=true;
>     dynamic_snitch_badness_threshold=0.1;
>     dynamic_snitch_reset_interval_in_ms=600000;
>     dynamic_snitch_update_interval_in_ms=100;
>     enable_scripted_user_defined_functions=false;
>     enable_user_defined_functions=false;
>     enable_user_defined_functions_threads=true;
>     encryption_options=null; endpoint_snitch=SimpleSnitch;
>     file_cache_size_in_mb=null; gc_log_threshold_in_ms=200;
>     gc_warn_threshold_in_ms=1000;
>     hinted_handoff_disabled_datacenters=[];
>     hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
>     hints_compression=null; hints_directory=/mnt/cassandra/hints;
>     hints_flush_period_in_ms=10000; incremental_backups=false;
>     index_interval=null; index_summary_capacity_in_mb=null;
>     index_summary_resize_interval_in_minutes=60; initial_token=null;
>     inter_dc_stream_throughput_outbound_megabits_per_sec=200;
>     inter_dc_tcp_nodelay=false; internode_authenticator=null;
>     internode_compression=dc; internode_recv_buff_size_in_bytes=0;
>     internode_send_buff_size_in_bytes=0;
>     key_cache_keys_to_save=2147483647; key_cache_save_period=14400;
>     key_cache_size_in_mb=null; listen_address=10.100.100.213;
>     listen_interface=null; listen_interface_prefer_ipv6=false;
>     listen_on_broadcast_address=false; max_hint_window_in_ms=10800000;
>     max_hints_delivery_threads=2; max_hints_file_size_in_mb=128;
>     max_mutation_size_in_kb=null; max_streaming_retries=3;
>     max_value_size_in_mb=256; memtable_allocation_type=heap_buffers;
>     memtable_cleanup_threshold=null; memtable_flush_writers=0;
>     memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null;
>     min_free_space_per_drive_in_mb=50;
>     native_transport_max_concurrent_connections=-1;
>     native_transport_max_concurrent_connections_per_ip=-1;
>     native_transport_max_frame_size_in_mb=256;
>     native_transport_max_threads=128; native_transport_port=9042;
>     native_transport_port_ssl=null; num_tokens=256;
>     otc_coalescing_strategy=TIMEHORIZON; otc_coalescing_window_us=200;
>     partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
>     permissions_cache_max_entries=1000;
>     permissions_update_interval_in_ms=-1;
>     permissions_validity_in_ms=2000; phi_convict_threshold=8.0;
>     prepared_statements_cache_size_mb=null;
>     range_request_timeout_in_ms=6000000;
>     read_request_timeout_in_ms=6000000;
>     request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
>     request_scheduler_id=null; request_scheduler_options=null;
>     request_timeout_in_ms=6000000; role_manager=CassandraRoleManager;
>     roles_cache_max_entries=1000; roles_update_interval_in_ms=-1;
>     roles_validity_in_ms=2000;
>     row_cache_class_name=org.apache.cassandra.cache.OHCProvider;
>     row_cache_keys_to_save=2147483647; row_cache_save_period=0;
>     row_cache_size_in_mb=0; rpc_address=10.100.100.213;
>     rpc_interface=null; rpc_interface_prefer_ipv6=false;
>     rpc_keepalive=true; rpc_listen_backlog=50;
>     rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160;
>     rpc_recv_buff_size_in_bytes=null;
>     rpc_send_buff_size_in_bytes=null; rpc_server_type=sync;
>     saved_caches_directory=/mnt/cassandra/saved_caches;
>     seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=10.100.100.19,
>     10.100.100.85, 10.100.100.185, 10.100.100.161, 10.100.100.52,
>     10.100.1000.213}; server_encryption_options=<REDACTED>;
>     slow_query_log_timeout_in_ms=6000000;
>     snapshot_before_compaction=false; ssl_storage_port=7001;
>     sstable_preemptive_open_interval_in_mb=50;
>     start_native_transport=true; start_rpc=false; storage_port=7000;
>     stream_throughput_outbound_megabits_per_sec=200;
>     streaming_keep_alive_period_in_secs=300;
>     streaming_socket_timeout_in_ms=86400000;
>     thrift_framed_transport_size_in_mb=15;
>     thrift_max_message_length_in_mb=16;
>     thrift_prepared_statements_cache_size_mb=null;
>     tombstone_failure_threshold=100000; tombstone_warn_threshold=1000;
>     tracetype_query_ttl=86400; tracetype_repair_ttl=604800;
>     transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@38c5cc4c;
>     trickle_fsync=false; trickle_fsync_interval_in_kb=10240;
>     truncate_request_timeout_in_ms=6000000;
>     unlogged_batch_across_partitions_warn_threshold=10;
>     user_defined_function_fail_timeout=1500;
>     user_defined_function_warn_timeout=500;
>     user_function_timeout_policy=die; windows_timer_interval=1;
>     write_request_timeout_in_ms=6000000]
>
>     Thanks
>
>
>     On 04/06/2017 11:30 AM, benjamin roth wrote:
>>     Have you checked the effective limits of a running CS process?
>>     Is CS run as Cassandra? Just to rule out missing file perms. 
>>
>>
>>     Am 06.04.2017 12:24 schrieb "Cogumelos Maravilha"
>>     <cogumelosmaravilha@sapo.pt <mailto:cogumelosmaravilha@sapo.pt>>:
>>
>>         From cassandra.yaml:
>>
>>         hints_directory: /mnt/cassandra/hints
>>         data_file_directories:
>>             - /mnt/cassandra/data
>>         commitlog_directory: /mnt/cassandra/commitlog
>>         saved_caches_directory: /mnt/cassandra/saved_caches
>>
>>         drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/
>>
>>         drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
>>         drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
>>         drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
>>         drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
>>         drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
>>         drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17 saved_caches/
>>
>>         cassand+  2267     1 99 10:18 ?        00:02:56 java
>>         -Xloggc:/var/log/cassandra/gc.log -ea
>>         -XX:+UseThreadPriorities -XX:Threa...
>>
>>         /dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt
>>
>>         On /etc/security/limits.conf
>>
>>         *           -       memlock      unlimited
>>         *           -      nofile      100000
>>         *           -      nproc          32768
>>         *           -      as           unlimited
>>
>>         On /etc/security/limits.d/cassandra.conf
>>
>>         cassandra  -  memlock  unlimited
>>         cassandra  -  nofile   100000
>>         cassandra  -  as       unlimited
>>         cassandra  -  nproc    32768
>>
>>         On /etc/sysctl.conf
>>
>>         vm.max_map_count = 1048575
>>
>>         On /etc/systcl.d/cassanda.conf
>>
>>         vm.max_map_count = 1048575
>>         net.ipv4.tcp_keepalive_time=600
>>
>>         On /etc/pam.d/su
>>         ...
>>         session    required   pam_limits.so
>>         ...
>>
>>         Distro is the currently Ubuntu LTS.
>>         Thanks
>>
>>
>>
>>         On 04/06/2017 10:39 AM, benjamin roth wrote:
>>>         Cassandra cannot write an SSTable to disk. Are you sure the
>>>         disk/volume where SSTables reside (normally
>>>         /var/lib/cassandra/data) is writeable for the CS user and
>>>         has enough free space?
>>>         The CDC warning also implies that.
>>>         The other warnings indicate you are probably not running CS
>>>         as root and you did not set an appropriate limit for max
>>>         open files. Running out of open files can also be a reason
>>>         for the IO error.
>>>
>>>         2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
>>>         <cogumelosmaravilha@sapo.pt
>>>         <mailto:cogumelosmaravilha@sapo.pt>>:
>>>
>>>             Hi list,
>>>
>>>             I'm using C* 3.10 in a 6 nodes cluster RF=2. All
>>>             instances type
>>>             i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS
>>>             formated 885G. I have
>>>             one node that is always dieing and I don't understand
>>>             why. Can anyone
>>>             give me some hints please. All nodes using the same
>>>             configuration.
>>>
>>>             Thanks in advance.
>>>
>>>             INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
>>>             IndexSummaryRedistribution.java:75 - Redistributing
>>>             index summaries
>>>             ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
>>>             CassandraDaemon.java:229 - Exception in thread
>>>             Thread[MemtablePostFlush:22,5,main]
>>>             org.apache.cassandra.io
>>>             <http://org.apache.cassandra.io>.FSWriteError:
>>>             java.io.IOException: Input/output
>>>             error
>>>                 at
>>>             org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io
>>>             <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io
>>>             <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter$TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(SequentialWriter.java:358)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io
>>>             <http://org.apache.cassandra.io>.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:367)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io
>>>             <http://org.apache.cassandra.io>.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io
>>>             <http://org.apache.cassandra.io>.sstable.SimpleSSTableMultiWriter.prepareToCommit(SimpleSSTableMultiWriter.java:101)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1153)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1086)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>             ~[na:1.8.0_121]
>>>                 at
>>>             java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>             [na:1.8.0_121]
>>>                 at
>>>             org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
>>>             [apache-cassandra-3.10.jar:3.10]
>>>                 at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
>>>             Caused by: java.io.IOException: Input/output error
>>>                 at sun.nio.ch.FileDispatcherImpl.force0(Native
>>>             Method) ~[na:1.8.0_121]
>>>                 at
>>>             sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
>>>             ~[na:1.8.0_121]
>>>                 at
>>>             sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388)
>>>             ~[na:1.8.0_121]
>>>                 at
>>>             org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:169)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 ... 15 common frames omitted
>>>             INFO  [IndexSummaryManager:1] 2017-04-06 06:22:18,366
>>>             IndexSummaryRedistribution.java:75 - Redistributing
>>>             index summaries
>>>             ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525
>>>             CassandraDaemon.java:229 - Exception in thread
>>>             Thread[MemtablePostFlush:31,5,main]
>>>             org.apache.cassandra.io
>>>             <http://org.apache.cassandra.io>.FSWriteError:
>>>             java.io.IOException: Input/output
>>>             error
>>>                 at
>>>             org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io
>>>             <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io
>>>             <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter$TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(SequentialWriter.java:358)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io
>>>             <http://org.apache.cassandra.io>.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:367)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io
>>>             <http://org.apache.cassandra.io>.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io
>>>             <http://org.apache.cassandra.io>.sstable.SimpleSSTableMultiWriter.prepareToCommit(SimpleSSTableMultiWriter.java:101)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1153)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1086)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>             ~[na:1.8.0_121]
>>>                 at
>>>             java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>             [na:1.8.0_121]
>>>                 at
>>>             org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
>>>             [apache-cassandra-3.10.jar:3.10]
>>>                 at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
>>>             Caused by: java.io.IOException: Input/output error
>>>                 at sun.nio.ch.FileDispatcherImpl.force0(Native
>>>             Method) ~[na:1.8.0_121]
>>>                 at
>>>             sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
>>>             ~[na:1.8.0_121]
>>>                 at
>>>             sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388)
>>>             ~[na:1.8.0_121]
>>>                 at
>>>             org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 at
>>>             org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:169)
>>>             ~[apache-cassandra-3.10.jar:3.10]
>>>                 ... 15 common frames omitted
>>>             INFO  [main] 2017-04-06 07:11:57,289
>>>             YamlConfigurationLoader.java:89 -
>>>             Configuration location: file:/etc/cassandra/cassandra.yaml
>>>
>>>
>>>             Some ERRORs messages:
>>>
>>>             ERROR [MemtablePostFlush:2] 2017-04-05 23:35:46,339
>>>             CassandraDaemon.java:229 - Exception in thread
>>>             Thread[MemtablePostFlush:2,5,main]
>>>             ERROR [MemtablePostFlush:3] 2017-04-05 23:44:08,471
>>>             CassandraDaemon.java:229 - Exception in thread
>>>             Thread[MemtablePostFlush:3,5,main]
>>>             ERROR [MemtablePostFlush:4] 2017-04-05 23:54:41,224
>>>             CassandraDaemon.java:229 - Exception in thread
>>>             Thread[MemtablePostFlush:4,5,main]
>>>             ERROR [MessagingService-Incoming-/10.0.120.52
>>>             <http://10.0.120.52>] 2017-04-06 03:19:13,453
>>>             CassandraDaemon.java:229 - Exception in thread
>>>             Thread[MessagingService-Incoming-/10.0.120.52
>>>             <http://10.0.120.52>,5,main]
>>>             ERROR [epollEventLoopGroup-2-6] 2017-04-06 03:24:41,006
>>>             CassandraDaemon.java:229 - Exception in thread
>>>             Thread[epollEventLoopGroup-2-6,10,main]
>>>             ERROR [Native-Transport-Requests-36] 2017-04-06 03:25:45,915
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [Native-Transport-Requests-49] 2017-04-06 03:25:45,915
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [IndexSummaryManager:1] 2017-04-06 03:25:45,915
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [Native-Transport-Requests-69] 2017-04-06 03:25:45,916
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [Native-Transport-Requests-46] 2017-04-06 03:26:18,465
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [SharedPool-Worker-136] 2017-04-06 03:26:18,465
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [Native-Transport-Requests-156] 2017-04-06
>>>             03:26:18,465
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [SharedPool-Worker-92] 2017-04-06 03:26:24,696
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [Native-Transport-Requests-48] 2017-04-06
>>>             03:26:24,696 ?:? - JVM
>>>             state determined to be unstable.  Exiting forcefully due to:
>>>             ERROR [Native-Transport-Requests-66] 2017-04-06 03:26:55,808
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [Native-Transport-Requests-77] 2017-04-06 03:26:55,808
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [GossipTasks:1] 2017-04-06 03:26:55,808
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [Native-Transport-Requests-133] 2017-04-06
>>>             03:26:55,808
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [Native-Transport-Requests-135] 2017-04-06
>>>             03:26:55,808
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [ScheduledFastTasks:1] 2017-04-06 03:26:55,808
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [Native-Transport-Requests-70] 2017-04-06 03:27:11,569
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [IndexSummaryManager:1] 2017-04-06 03:27:17,821
>>>             CassandraDaemon.java:229 - Exception in thread
>>>             Thread[IndexSummaryManager:1,1,main]
>>>             ERROR [Native-Transport-Requests-103] 2017-04-06
>>>             03:27:24,049
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [Native-Transport-Requests-69] 2017-04-06 03:27:24,049
>>>             SEPWorker.java:145 - Failed to execute task, unexpected
>>>             exception killed
>>>             worker: {}
>>>             ERROR [SharedPool-Worker-98] 2017-04-06 03:27:24,049
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [MessagingService-Incoming-/10.0.120.52
>>>             <http://10.0.120.52>] 2017-04-06 03:27:55,079
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [epollEventLoopGroup-2-5] 2017-04-06 03:27:55,079
>>>             JVMStabilityInspector.java:142 - JVM state determined to
>>>             be unstable.
>>>             Exiting forcefully due to:
>>>             ERROR [Native-Transport-Requests-64] 2017-04-06 03:28:43,285
>>>             SEPWorker.java:145 - Failed to execute task, unexpected
>>>             exception killed
>>>             worker: {}
>>>             ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
>>>             CassandraDaemon.java:229 - Exception in thread
>>>             Thread[MemtablePostFlush:22,5,main]
>>>             ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525
>>>             CassandraDaemon.java:229 - Exception in thread
>>>             Thread[MemtablePostFlush:31,5,main]
>>>
>>>             Also some WARNs:
>>>
>>>             WARN  [main] 2017-04-06 09:26:49,725 CLibrary.java:178 -
>>>             Unable to lock
>>>             JVM memory (ENOMEM). This can result in part of the JVM
>>>             being swapped
>>>             out, especially with mmapped I/O enabled. Increase
>>>             RLIMIT_MEMLOCK or run
>>>             Cassandra as root.
>>>
>>>             WARN  [main] 2017-04-06 09:25:07,355
>>>             StartupChecks.java:157 - JMX is not
>>>             enabled to receive remote connections. Please see
>>>             cassandra-env.sh for
>>>             more info.
>>>
>>>             WARN  [main] 2017-04-06 09:25:07,369
>>>             SigarLibrary.java:174 - Cassandra
>>>             server running in degraded mode. Is swap disabled? :
>>>             true,  Address
>>>             space adequate? : true,  nofile limit adequate? : false,
>>>             nproc limit
>>>             adequate? : true
>>>
>>>             WARN  [main] 2017-04-06 09:25:07,091
>>>             DatabaseDescriptor.java:493 - Small
>>>             cdc volume detected at /var/lib/cassandra/cdc_raw; setting
>>>             cdc_total_space_in_mb to 2502.  You can override this in
>>>             cassandra.yaml
>>>
>>>
>>>
>>
>>
>
>


Mime
View raw message