cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cogumelos Maravilha <cogumelosmaravi...@sapo.pt>
Subject Re: Node always dieing
Date Thu, 06 Apr 2017 13:08:26 GMT
We tested in c4 instances but EBS is too slow. So we deployed for
production in i3.

It was running with 5 nodes without problems but we started running out
of space so we added another node. And is this last node that is giving
problems. I've already terminated the instance and created another once,
but the problem remains. The configuration is the same for all
instances. Runs nice for a few time but then

ERROR [MemtablePostFlush:11] 2017-04-06 11:47:40,840
CassandraDaemon.java:229 - Exception in thread
Thread[MemtablePostFlush:11,5,main]

Thanks.

On 04/06/2017 01:18 PM, Carlos Rolo wrote:
> There was some issue with the i3 instances and Cassandra. Did you had
> this cluster running always on i3? 
>
> On Apr 6, 2017 13:06, "Cogumelos Maravilha"
> <cogumelosmaravilha@sapo.pt <mailto:cogumelosmaravilha@sapo.pt>> wrote:
>
>     Limit                     Soft Limit           Hard
>     Limit           Units    
>     Max cpu time              unlimited           
>     unlimited            seconds  
>     Max file size             unlimited           
>     unlimited            bytes    
>     Max data size             unlimited           
>     unlimited            bytes    
>     Max stack size            8388608             
>     unlimited            bytes    
>     Max core file size        0                   
>     unlimited            bytes    
>     Max resident set          unlimited           
>     unlimited            bytes    
>     Max processes             122575              
>     122575               processes
>     Max open files            100000              
>     100000               files    
>     Max locked memory         unlimited           
>     unlimited            bytes    
>     Max address space         unlimited           
>     unlimited            bytes    
>     Max file locks            unlimited           
>     unlimited            locks    
>     Max pending signals       122575              
>     122575               signals  
>     Max msgqueue size         819200              
>     819200               bytes    
>     Max nice priority         0                    0                   
>     Max realtime priority     0                    0                   
>     Max realtime timeout      unlimited           
>     unlimited            us   
>
>     Please find something wrong there!
>
>     Thanks.
>
>     On 04/06/2017 11:50 AM, benjamin roth wrote:
>>     Limits: You should check them in /proc/$pid/limits
>>
>>     2017-04-06 12:48 GMT+02:00 Cogumelos Maravilha
>>     <cogumelosmaravilha@sapo.pt <mailto:cogumelosmaravilha@sapo.pt>>:
>>
>>         Yes C* is running as cassandra:
>>
>>         cassand+  2267     1 99 10:18 ?        00:02:56 java
>>         -Xloggc:/var/log/cassandra/gc.log -ea
>>         -XX:+UseThreadPriorities -XX:Threa...
>>
>>         INFO  [main] 2017-04-06 10:35:42,956 Config.java:474 - Node
>>         configuration:[allocate_tokens_for_keyspace=null;
>>         authenticator=PasswordAuthenticator;
>>         authorizer=CassandraAuthorizer; auto_bootstrap=true;
>>         auto_snapshot=true; back_pressure_enabled=false;
>>         back_pressure_strategy=org.apache.cassandra.net
>>         <http://apache.cassandra.net>.RateBasedBackPressure{high_ratio=0.9,
>>         factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
>>         batch_size_warn_threshold_in_kb=5;
>>         batchlog_replay_throttle_in_kb=1024; broadcast_address=null;
>>         broadcast_rpc_address=null;
>>         buffer_pool_use_heap_if_exhausted=true;
>>         cas_contention_timeout_in_ms=6000000; cdc_enabled=false;
>>         cdc_free_space_check_interval_ms=250; cdc_raw_directory=null;
>>         cdc_total_space_in_mb=0;
>>         client_encryption_options=<REDACTED>; cluster_name=company;
>>         column_index_cache_size_in_kb=2; column_index_size_in_kb=64;
>>         commit_failure_policy=ignore; commitlog_compression=null;
>>         commitlog_directory=/mnt/cassandra/commitlog;
>>         commitlog_max_compression_buffers_in_pool=3;
>>         commitlog_periodic_queue_size=-1;
>>         commitlog_segment_size_in_mb=32; commitlog_sync=periodic;
>>         commitlog_sync_batch_window_in_ms=NaN;
>>         commitlog_sync_period_in_ms=10000;
>>         commitlog_total_space_in_mb=null;
>>         compaction_large_partition_warning_threshold_mb=100;
>>         compaction_throughput_mb_per_sec=16;
>>         concurrent_compactors=null; concurrent_counter_writes=32;
>>         concurrent_materialized_view_writes=32; concurrent_reads=32;
>>         concurrent_replicates=null; concurrent_writes=32;
>>         counter_cache_keys_to_save=2147483647;
>>         counter_cache_save_period=7200;
>>         counter_cache_size_in_mb=null;
>>         counter_write_request_timeout_in_ms=6000000;
>>         credentials_cache_max_entries=1000;
>>         credentials_update_interval_in_ms=-1;
>>         credentials_validity_in_ms=2000; cross_node_timeout=false;
>>         data_file_directories=[Ljava.lang.String;@223f3642;
>>         disk_access_mode=auto; disk_failure_policy=ignore;
>>         disk_optimization_estimate_percentile=0.95;
>>         disk_optimization_page_cross_chance=0.1;
>>         disk_optimization_strategy=ssd; dynamic_snitch=true;
>>         dynamic_snitch_badness_threshold=0.1;
>>         dynamic_snitch_reset_interval_in_ms=600000;
>>         dynamic_snitch_update_interval_in_ms=100;
>>         enable_scripted_user_defined_functions=false;
>>         enable_user_defined_functions=false;
>>         enable_user_defined_functions_threads=true;
>>         encryption_options=null; endpoint_snitch=SimpleSnitch;
>>         file_cache_size_in_mb=null; gc_log_threshold_in_ms=200;
>>         gc_warn_threshold_in_ms=1000;
>>         hinted_handoff_disabled_datacenters=[];
>>         hinted_handoff_enabled=true;
>>         hinted_handoff_throttle_in_kb=1024; hints_compression=null;
>>         hints_directory=/mnt/cassandra/hints;
>>         hints_flush_period_in_ms=10000; incremental_backups=false;
>>         index_interval=null; index_summary_capacity_in_mb=null;
>>         index_summary_resize_interval_in_minutes=60;
>>         initial_token=null;
>>         inter_dc_stream_throughput_outbound_megabits_per_sec=200;
>>         inter_dc_tcp_nodelay=false; internode_authenticator=null;
>>         internode_compression=dc;
>>         internode_recv_buff_size_in_bytes=0;
>>         internode_send_buff_size_in_bytes=0;
>>         key_cache_keys_to_save=2147483647;
>>         key_cache_save_period=14400; key_cache_size_in_mb=null;
>>         listen_address=10.100.100.213; listen_interface=null;
>>         listen_interface_prefer_ipv6=false;
>>         listen_on_broadcast_address=false;
>>         max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
>>         max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null;
>>         max_streaming_retries=3; max_value_size_in_mb=256;
>>         memtable_allocation_type=heap_buffers;
>>         memtable_cleanup_threshold=null; memtable_flush_writers=0;
>>         memtable_heap_space_in_mb=null;
>>         memtable_offheap_space_in_mb=null;
>>         min_free_space_per_drive_in_mb=50;
>>         native_transport_max_concurrent_connections=-1;
>>         native_transport_max_concurrent_connections_per_ip=-1;
>>         native_transport_max_frame_size_in_mb=256;
>>         native_transport_max_threads=128; native_transport_port=9042;
>>         native_transport_port_ssl=null; num_tokens=256;
>>         otc_coalescing_strategy=TIMEHORIZON;
>>         otc_coalescing_window_us=200;
>>         partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
>>         permissions_cache_max_entries=1000;
>>         permissions_update_interval_in_ms=-1;
>>         permissions_validity_in_ms=2000; phi_convict_threshold=8.0;
>>         prepared_statements_cache_size_mb=null;
>>         range_request_timeout_in_ms=6000000;
>>         read_request_timeout_in_ms=6000000;
>>         request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
>>         request_scheduler_id=null; request_scheduler_options=null;
>>         request_timeout_in_ms=6000000;
>>         role_manager=CassandraRoleManager;
>>         roles_cache_max_entries=1000; roles_update_interval_in_ms=-1;
>>         roles_validity_in_ms=2000;
>>         row_cache_class_name=org.apache.cassandra.cache.OHCProvider;
>>         row_cache_keys_to_save=2147483647; row_cache_save_period=0;
>>         row_cache_size_in_mb=0; rpc_address=10.100.100.213;
>>         rpc_interface=null; rpc_interface_prefer_ipv6=false;
>>         rpc_keepalive=true; rpc_listen_backlog=50;
>>         rpc_max_threads=2147483647; rpc_min_threads=16;
>>         rpc_port=9160; rpc_recv_buff_size_in_bytes=null;
>>         rpc_send_buff_size_in_bytes=null; rpc_server_type=sync;
>>         saved_caches_directory=/mnt/cassandra/saved_caches;
>>         seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=10.100.100.19,
>>         10.100.100.85, 10.100.100.185, 10.100.100.161, 10.100.100.52,
>>         10.100.1000.213}; server_encryption_options=<REDACTED>;
>>         slow_query_log_timeout_in_ms=6000000;
>>         snapshot_before_compaction=false; ssl_storage_port=7001;
>>         sstable_preemptive_open_interval_in_mb=50;
>>         start_native_transport=true; start_rpc=false;
>>         storage_port=7000;
>>         stream_throughput_outbound_megabits_per_sec=200;
>>         streaming_keep_alive_period_in_secs=300;
>>         streaming_socket_timeout_in_ms=86400000;
>>         thrift_framed_transport_size_in_mb=15;
>>         thrift_max_message_length_in_mb=16;
>>         thrift_prepared_statements_cache_size_mb=null;
>>         tombstone_failure_threshold=100000;
>>         tombstone_warn_threshold=1000; tracetype_query_ttl=86400;
>>         tracetype_repair_ttl=604800;
>>         transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@38c5cc4c;
>>         trickle_fsync=false; trickle_fsync_interval_in_kb=10240;
>>         truncate_request_timeout_in_ms=6000000;
>>         unlogged_batch_across_partitions_warn_threshold=10;
>>         user_defined_function_fail_timeout=1500;
>>         user_defined_function_warn_timeout=500;
>>         user_function_timeout_policy=die; windows_timer_interval=1;
>>         write_request_timeout_in_ms=6000000]
>>
>>         Thanks
>>
>>
>>         On 04/06/2017 11:30 AM, benjamin roth wrote:
>>>         Have you checked the effective limits of a running CS process?
>>>         Is CS run as Cassandra? Just to rule out missing file perms. 
>>>
>>>
>>>         Am 06.04.2017 12:24 schrieb "Cogumelos Maravilha"
>>>         <cogumelosmaravilha@sapo.pt
>>>         <mailto:cogumelosmaravilha@sapo.pt>>:
>>>
>>>             From cassandra.yaml:
>>>
>>>             hints_directory: /mnt/cassandra/hints
>>>             data_file_directories:
>>>                 - /mnt/cassandra/data
>>>             commitlog_directory: /mnt/cassandra/commitlog
>>>             saved_caches_directory: /mnt/cassandra/saved_caches
>>>
>>>             drwxr-xr-x   3 cassandra cassandra   23 Apr  5 16:03 mnt/
>>>
>>>             drwxr-xr-x 6 cassandra cassandra  68 Apr  5 16:17 ./
>>>             drwxr-xr-x 3 cassandra cassandra  23 Apr  5 16:03 ../
>>>             drwxr-xr-x 2 cassandra cassandra  80 Apr  6 10:07 commitlog/
>>>             drwxr-xr-x 8 cassandra cassandra 124 Apr  5 16:17 data/
>>>             drwxr-xr-x 2 cassandra cassandra  72 Apr  5 16:20 hints/
>>>             drwxr-xr-x 2 cassandra cassandra  49 Apr  5 20:17
>>>             saved_caches/
>>>
>>>             cassand+  2267     1 99 10:18 ?        00:02:56 java
>>>             -Xloggc:/var/log/cassandra/gc.log -ea
>>>             -XX:+UseThreadPriorities -XX:Threa...
>>>
>>>             /dev/mapper/um_vg-xfs_lv  885G   27G  858G   4% /mnt
>>>
>>>             On /etc/security/limits.conf
>>>
>>>             *           -       memlock      unlimited
>>>             *           -      nofile      100000
>>>             *           -      nproc          32768
>>>             *           -      as           unlimited
>>>
>>>             On /etc/security/limits.d/cassandra.conf
>>>
>>>             cassandra  -  memlock  unlimited
>>>             cassandra  -  nofile   100000
>>>             cassandra  -  as       unlimited
>>>             cassandra  -  nproc    32768
>>>
>>>             On /etc/sysctl.conf
>>>
>>>             vm.max_map_count = 1048575
>>>
>>>             On /etc/systcl.d/cassanda.conf
>>>
>>>             vm.max_map_count = 1048575
>>>             net.ipv4.tcp_keepalive_time=600
>>>
>>>             On /etc/pam.d/su
>>>             ...
>>>             session    required   pam_limits.so
>>>             ...
>>>
>>>             Distro is the currently Ubuntu LTS.
>>>             Thanks
>>>
>>>
>>>
>>>             On 04/06/2017 10:39 AM, benjamin roth wrote:
>>>>             Cassandra cannot write an SSTable to disk. Are you sure
>>>>             the disk/volume where SSTables reside (normally
>>>>             /var/lib/cassandra/data) is writeable for the CS user
>>>>             and has enough free space?
>>>>             The CDC warning also implies that.
>>>>             The other warnings indicate you are probably not
>>>>             running CS as root and you did not set an appropriate
>>>>             limit for max open files. Running out of open files can
>>>>             also be a reason for the IO error.
>>>>
>>>>             2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha
>>>>             <cogumelosmaravilha@sapo.pt
>>>>             <mailto:cogumelosmaravilha@sapo.pt>>:
>>>>
>>>>                 Hi list,
>>>>
>>>>                 I'm using C* 3.10 in a 6 nodes cluster RF=2. All
>>>>                 instances type
>>>>                 i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS
>>>>                 formated 885G. I have
>>>>                 one node that is always dieing and I don't
>>>>                 understand why. Can anyone
>>>>                 give me some hints please. All nodes using the same
>>>>                 configuration.
>>>>
>>>>                 Thanks in advance.
>>>>
>>>>                 INFO  [IndexSummaryManager:1] 2017-04-06 05:22:18,352
>>>>                 IndexSummaryRedistribution.java:75 - Redistributing
>>>>                 index summaries
>>>>                 ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
>>>>                 CassandraDaemon.java:229 - Exception in thread
>>>>                 Thread[MemtablePostFlush:22,5,main]
>>>>                 org.apache.cassandra.io
>>>>                 <http://org.apache.cassandra.io>.FSWriteError:
>>>>                 java.io.IOException: Input/output
>>>>                 error
>>>>                     at
>>>>                 org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io
>>>>                 <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io
>>>>                 <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter$TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(SequentialWriter.java:358)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io
>>>>                 <http://org.apache.cassandra.io>.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:367)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io
>>>>                 <http://org.apache.cassandra.io>.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io
>>>>                 <http://org.apache.cassandra.io>.sstable.SimpleSSTableMultiWriter.prepareToCommit(SimpleSSTableMultiWriter.java:101)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1153)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1086)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>                 ~[na:1.8.0_121]
>>>>                     at
>>>>                 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>                 [na:1.8.0_121]
>>>>                     at
>>>>                 org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
>>>>                 [apache-cassandra-3.10.jar:3.10]
>>>>                     at java.lang.Thread.run(Thread.java:745)
>>>>                 ~[na:1.8.0_121]
>>>>                 Caused by: java.io.IOException: Input/output error
>>>>                     at sun.nio.ch.FileDispatcherImpl.force0(Native
>>>>                 Method) ~[na:1.8.0_121]
>>>>                     at
>>>>                 sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
>>>>                 ~[na:1.8.0_121]
>>>>                     at
>>>>                 sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388)
>>>>                 ~[na:1.8.0_121]
>>>>                     at
>>>>                 org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:169)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     ... 15 common frames omitted
>>>>                 INFO  [IndexSummaryManager:1] 2017-04-06 06:22:18,366
>>>>                 IndexSummaryRedistribution.java:75 - Redistributing
>>>>                 index summaries
>>>>                 ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525
>>>>                 CassandraDaemon.java:229 - Exception in thread
>>>>                 Thread[MemtablePostFlush:31,5,main]
>>>>                 org.apache.cassandra.io
>>>>                 <http://org.apache.cassandra.io>.FSWriteError:
>>>>                 java.io.IOException: Input/output
>>>>                 error
>>>>                     at
>>>>                 org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:173)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io.util.SequentialWriter.syncInternal(SequentialWriter.java:185)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io
>>>>                 <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter.access$100(CompressedSequentialWriter.java:38)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io
>>>>                 <http://org.apache.cassandra.io>.compress.CompressedSequentialWriter$TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io.util.SequentialWriter.prepareToCommit(SequentialWriter.java:358)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io
>>>>                 <http://org.apache.cassandra.io>.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:367)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io
>>>>                 <http://org.apache.cassandra.io>.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io
>>>>                 <http://org.apache.cassandra.io>.sstable.SimpleSSTableMultiWriter.prepareToCommit(SimpleSSTableMultiWriter.java:101)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1153)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1086)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>                 ~[na:1.8.0_121]
>>>>                     at
>>>>                 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>                 [na:1.8.0_121]
>>>>                     at
>>>>                 org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
>>>>                 [apache-cassandra-3.10.jar:3.10]
>>>>                     at java.lang.Thread.run(Thread.java:745)
>>>>                 ~[na:1.8.0_121]
>>>>                 Caused by: java.io.IOException: Input/output error
>>>>                     at sun.nio.ch.FileDispatcherImpl.force0(Native
>>>>                 Method) ~[na:1.8.0_121]
>>>>                     at
>>>>                 sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
>>>>                 ~[na:1.8.0_121]
>>>>                     at
>>>>                 sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388)
>>>>                 ~[na:1.8.0_121]
>>>>                     at
>>>>                 org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     at
>>>>                 org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyInternal(SequentialWriter.java:169)
>>>>                 ~[apache-cassandra-3.10.jar:3.10]
>>>>                     ... 15 common frames omitted
>>>>                 INFO  [main] 2017-04-06 07:11:57,289
>>>>                 YamlConfigurationLoader.java:89 -
>>>>                 Configuration location:
>>>>                 file:/etc/cassandra/cassandra.yaml
>>>>
>>>>
>>>>                 Some ERRORs messages:
>>>>
>>>>                 ERROR [MemtablePostFlush:2] 2017-04-05 23:35:46,339
>>>>                 CassandraDaemon.java:229 - Exception in thread
>>>>                 Thread[MemtablePostFlush:2,5,main]
>>>>                 ERROR [MemtablePostFlush:3] 2017-04-05 23:44:08,471
>>>>                 CassandraDaemon.java:229 - Exception in thread
>>>>                 Thread[MemtablePostFlush:3,5,main]
>>>>                 ERROR [MemtablePostFlush:4] 2017-04-05 23:54:41,224
>>>>                 CassandraDaemon.java:229 - Exception in thread
>>>>                 Thread[MemtablePostFlush:4,5,main]
>>>>                 ERROR [MessagingService-Incoming-/10.0.120.52
>>>>                 <http://10.0.120.52>] 2017-04-06 03:19:13,453
>>>>                 CassandraDaemon.java:229 - Exception in thread
>>>>                 Thread[MessagingService-Incoming-/10.0.120.52
>>>>                 <http://10.0.120.52>,5,main]
>>>>                 ERROR [epollEventLoopGroup-2-6] 2017-04-06 03:24:41,006
>>>>                 CassandraDaemon.java:229 - Exception in thread
>>>>                 Thread[epollEventLoopGroup-2-6,10,main]
>>>>                 ERROR [Native-Transport-Requests-36] 2017-04-06
>>>>                 03:25:45,915
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [Native-Transport-Requests-49] 2017-04-06
>>>>                 03:25:45,915
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [IndexSummaryManager:1] 2017-04-06 03:25:45,915
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [Native-Transport-Requests-69] 2017-04-06
>>>>                 03:25:45,916
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [Native-Transport-Requests-46] 2017-04-06
>>>>                 03:26:18,465
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [SharedPool-Worker-136] 2017-04-06 03:26:18,465
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [Native-Transport-Requests-156] 2017-04-06
>>>>                 03:26:18,465
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [SharedPool-Worker-92] 2017-04-06 03:26:24,696
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [Native-Transport-Requests-48] 2017-04-06
>>>>                 03:26:24,696 ?:? - JVM
>>>>                 state determined to be unstable.  Exiting
>>>>                 forcefully due to:
>>>>                 ERROR [Native-Transport-Requests-66] 2017-04-06
>>>>                 03:26:55,808
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [Native-Transport-Requests-77] 2017-04-06
>>>>                 03:26:55,808
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [GossipTasks:1] 2017-04-06 03:26:55,808
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [Native-Transport-Requests-133] 2017-04-06
>>>>                 03:26:55,808
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [Native-Transport-Requests-135] 2017-04-06
>>>>                 03:26:55,808
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [ScheduledFastTasks:1] 2017-04-06 03:26:55,808
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [Native-Transport-Requests-70] 2017-04-06
>>>>                 03:27:11,569
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [IndexSummaryManager:1] 2017-04-06 03:27:17,821
>>>>                 CassandraDaemon.java:229 - Exception in thread
>>>>                 Thread[IndexSummaryManager:1,1,main]
>>>>                 ERROR [Native-Transport-Requests-103] 2017-04-06
>>>>                 03:27:24,049
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [Native-Transport-Requests-69] 2017-04-06
>>>>                 03:27:24,049
>>>>                 SEPWorker.java:145 - Failed to execute task,
>>>>                 unexpected exception killed
>>>>                 worker: {}
>>>>                 ERROR [SharedPool-Worker-98] 2017-04-06 03:27:24,049
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [MessagingService-Incoming-/10.0.120.52
>>>>                 <http://10.0.120.52>] 2017-04-06 03:27:55,079
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [epollEventLoopGroup-2-5] 2017-04-06 03:27:55,079
>>>>                 JVMStabilityInspector.java:142 - JVM state
>>>>                 determined to be unstable.
>>>>                 Exiting forcefully due to:
>>>>                 ERROR [Native-Transport-Requests-64] 2017-04-06
>>>>                 03:28:43,285
>>>>                 SEPWorker.java:145 - Failed to execute task,
>>>>                 unexpected exception killed
>>>>                 worker: {}
>>>>                 ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
>>>>                 CassandraDaemon.java:229 - Exception in thread
>>>>                 Thread[MemtablePostFlush:22,5,main]
>>>>                 ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525
>>>>                 CassandraDaemon.java:229 - Exception in thread
>>>>                 Thread[MemtablePostFlush:31,5,main]
>>>>
>>>>                 Also some WARNs:
>>>>
>>>>                 WARN  [main] 2017-04-06 09:26:49,725
>>>>                 CLibrary.java:178 - Unable to lock
>>>>                 JVM memory (ENOMEM). This can result in part of the
>>>>                 JVM being swapped
>>>>                 out, especially with mmapped I/O enabled. Increase
>>>>                 RLIMIT_MEMLOCK or run
>>>>                 Cassandra as root.
>>>>
>>>>                 WARN  [main] 2017-04-06 09:25:07,355
>>>>                 StartupChecks.java:157 - JMX is not
>>>>                 enabled to receive remote connections. Please see
>>>>                 cassandra-env.sh for
>>>>                 more info.
>>>>
>>>>                 WARN  [main] 2017-04-06 09:25:07,369
>>>>                 SigarLibrary.java:174 - Cassandra
>>>>                 server running in degraded mode. Is swap disabled?
>>>>                 : true,  Address
>>>>                 space adequate? : true,  nofile limit adequate? :
>>>>                 false, nproc limit
>>>>                 adequate? : true
>>>>
>>>>                 WARN  [main] 2017-04-06 09:25:07,091
>>>>                 DatabaseDescriptor.java:493 - Small
>>>>                 cdc volume detected at /var/lib/cassandra/cdc_raw;
>>>>                 setting
>>>>                 cdc_total_space_in_mb to 2502.  You can override
>>>>                 this in cassandra.yaml
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
> --
>
>
>


Mime
View raw message