Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id D94D9200C52 for ; Mon, 10 Apr 2017 13:37:24 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id D7D2A160B99; Mon, 10 Apr 2017 11:37:24 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0EF7E160B85 for ; Mon, 10 Apr 2017 13:37:22 +0200 (CEST) Received: (qmail 45012 invoked by uid 500); 10 Apr 2017 11:37:21 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 45002 invoked by uid 99); 10 Apr 2017 11:37:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Apr 2017 11:37:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 0F708CA7B4 for ; Mon, 10 Apr 2017 11:37:19 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.265 X-Spam-Level: * X-Spam-Status: No, score=1.265 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=2, HTML_OBFUSCATE_10_20=1.162, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.796, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id E9eE9s7703Te for ; Mon, 10 Apr 2017 11:37:15 +0000 (UTC) Received: from mail-oi0-f44.google.com (mail-oi0-f44.google.com [209.85.218.44]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 305B15FBF2 for ; Mon, 10 Apr 2017 11:37:14 +0000 (UTC) Received: by mail-oi0-f44.google.com with SMTP id b187so144293166oif.0 for ; Mon, 10 Apr 2017 04:37:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=nML4iwc7dkHBSB1Z2+yDlwBe0kmMO4FRx1d+GNb3LCs=; b=BwwtpX/+gs+SoLEoGyNC28qEHYAvDf0bx6Yb0Km8pbmzXZCEXzpreW9jdH7R2r7ZmC R3ne9j7FlnkNPa/pjg9DXazqE3lU6bnzX9b4709yBqS35ZjIJsKW1eGJ/kD3EXVvCvOc AFTioawLFRyejs7w3kjvRa8R08FgL+/Z6N2QCtYB6yptehX6xYScGw3cYq/KQc4dnC6t 1Nvkgv/wdfRKZ8OSgYYFe2y4vpBL5RAXWMc8goQaUo7MuY9Fz4N7lg9jFjjPCFGEOKtK ude1D17E5QEj7cw3NfPRcKPFzIaUbqnEuNCy82j7fv5/oM0b6Sl5V9sqqemi32vDGRAE m0Yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=nML4iwc7dkHBSB1Z2+yDlwBe0kmMO4FRx1d+GNb3LCs=; b=HwATeKaxCQDhFU4jOiW91+yu72VYgWVee7QMVUDkhcZ0kpgEO+aguAym+0M1/gUCA2 3NxQBQ42Ev/t2kpqpdyjMdozLA/dpeyJ2gcfhX4svhhu5vws4Ae8gbq6DU+Ys0ntG1ae vi3V7zrgUaseKciYrS03IFjuDVMO+jargcSdkIGqdsCF73wBz3a63DFiHmL54egLmG2U sO8YPkyTJ96GHRV3AUjl/JNE0e/+NwT/q4OYGzMFjoBskrylIh8dKPhm6h6ud1FXuiEI 8Kk1sBLl7gTjO6QLZeVtqbDbBKkkgKyHO4R5wrE3lo+t64SV7BGORwEeFN30Dxxn800D W6ig== X-Gm-Message-State: AN3rC/75iRnUOY6cSbJCW2K5ZQbuu4ceuSgs1T0e7+e8fhrZ/yYxH/kUrNbNikiQfzrC62ITh19W6G0YXFMZCQ== X-Received: by 10.157.38.248 with SMTP id i53mr1444361otd.55.1491824232662; Mon, 10 Apr 2017 04:37:12 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.52.195 with HTTP; Mon, 10 Apr 2017 04:37:12 -0700 (PDT) In-Reply-To: <3dd241af-6536-7e65-6c00-cacc5059cf82@sapo.pt> References: <3dd241af-6536-7e65-6c00-cacc5059cf82@sapo.pt> From: Chris Mawata Date: Mon, 10 Apr 2017 07:37:12 -0400 Message-ID: Subject: Re: Node always dieing To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a113e20b6748ceb054cce6589 archived-at: Mon, 10 Apr 2017 11:37:25 -0000 --001a113e20b6748ceb054cce6589 Content-Type: text/plain; charset=UTF-8 Notice .SimpleSeedProvider{seeds=10.100.100.19, 10.100.100.85, 10.100.100.185, 10.100.100.161, 10.100.100.52, 10.100.1000.213}; Why do you have all six of your nodes as seeds? is it possible that the last one you added used itself as the seed and is isolated? On Thu, Apr 6, 2017 at 6:48 AM, Cogumelos Maravilha < cogumelosmaravilha@sapo.pt> wrote: > Yes C* is running as cassandra: > > cassand+ 2267 1 99 10:18 ? 00:02:56 java > -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities > -XX:Threa... > > INFO [main] 2017-04-06 10:35:42,956 Config.java:474 - Node > configuration:[allocate_tokens_for_keyspace=null; authenticator=PasswordAuthenticator; > authorizer=CassandraAuthorizer; auto_bootstrap=true; auto_snapshot=true; > back_pressure_enabled=false; back_pressure_strategy=org. > apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, > flow=FAST}; batch_size_fail_threshold_in_kb=50; > batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; > broadcast_address=null; broadcast_rpc_address=null; buffer_pool_use_heap_if_exhausted=true; > cas_contention_timeout_in_ms=6000000; cdc_enabled=false; > cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; > cdc_total_space_in_mb=0; client_encryption_options=; > cluster_name=company; column_index_cache_size_in_kb=2; > column_index_size_in_kb=64; commit_failure_policy=ignore; > commitlog_compression=null; commitlog_directory=/mnt/cassandra/commitlog; > commitlog_max_compression_buffers_in_pool=3; > commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; > commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; > commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; > compaction_large_partition_warning_threshold_mb=100; > compaction_throughput_mb_per_sec=16; concurrent_compactors=null; > concurrent_counter_writes=32; concurrent_materialized_view_writes=32; > concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; > counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; > counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=6000000; > credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; > credentials_validity_in_ms=2000; cross_node_timeout=false; > data_file_directories=[Ljava.lang.String;@223f3642; > disk_access_mode=auto; disk_failure_policy=ignore; > disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; > disk_optimization_strategy=ssd; dynamic_snitch=true; > dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; > dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; > enable_user_defined_functions=false; enable_user_defined_functions_threads=true; > encryption_options=null; endpoint_snitch=SimpleSnitch; > file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; > gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; > hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; > hints_compression=null; hints_directory=/mnt/cassandra/hints; > hints_flush_period_in_ms=10000; incremental_backups=false; > index_interval=null; index_summary_capacity_in_mb=null; > index_summary_resize_interval_in_minutes=60; initial_token=null; > inter_dc_stream_throughput_outbound_megabits_per_sec=200; > inter_dc_tcp_nodelay=false; internode_authenticator=null; > internode_compression=dc; internode_recv_buff_size_in_bytes=0; > internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; > key_cache_save_period=14400; key_cache_size_in_mb=null; > listen_address=10.100.100.213; listen_interface=null; > listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; > max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; > max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; > max_streaming_retries=3; max_value_size_in_mb=256; > memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; > memtable_flush_writers=0; memtable_heap_space_in_mb=null; > memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; > native_transport_max_concurrent_connections=-1; native_transport_max_ > concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; > native_transport_max_threads=128; native_transport_port=9042; > native_transport_port_ssl=null; num_tokens=256; otc_coalescing_strategy=TIMEHORIZON; > otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; > permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; > permissions_validity_in_ms=2000; phi_convict_threshold=8.0; > prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=6000000; > read_request_timeout_in_ms=6000000; request_scheduler=org.apache. > cassandra.scheduler.NoScheduler; request_scheduler_id=null; > request_scheduler_options=null; request_timeout_in_ms=6000000; > role_manager=CassandraRoleManager; roles_cache_max_entries=1000; > roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; > row_cache_class_name=org.apache.cassandra.cache.OHCProvider; > row_cache_keys_to_save=2147483647; row_cache_save_period=0; > row_cache_size_in_mb=0; rpc_address=10.100.100.213; rpc_interface=null; > rpc_interface_prefer_ipv6=false; rpc_keepalive=true; > rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; > rpc_port=9160; rpc_recv_buff_size_in_bytes=null; > rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; > saved_caches_directory=/mnt/cassandra/saved_caches; > seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=10.100.100.19, > 10.100.100.85, 10.100.100.185, 10.100.100.161, 10.100.100.52, > 10.100.1000.213}; server_encryption_options=; > slow_query_log_timeout_in_ms=6000000; snapshot_before_compaction=false; > ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; > start_native_transport=true; start_rpc=false; storage_port=7000; > stream_throughput_outbound_megabits_per_sec=200; > streaming_keep_alive_period_in_secs=300; streaming_socket_timeout_in_ms=86400000; > thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; > thrift_prepared_statements_cache_size_mb=null; > tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; > tracetype_query_ttl=86400; tracetype_repair_ttl=604800; > transparent_data_encryption_options=org.apache.cassandra.config. > TransparentDataEncryptionOptions@38c5cc4c; trickle_fsync=false; > trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=6000000; > unlogged_batch_across_partitions_warn_threshold=10; > user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; > user_function_timeout_policy=die; windows_timer_interval=1; > write_request_timeout_in_ms=6000000] > Thanks > > > On 04/06/2017 11:30 AM, benjamin roth wrote: > > Have you checked the effective limits of a running CS process? > Is CS run as Cassandra? Just to rule out missing file perms. > > > Am 06.04.2017 12:24 schrieb "Cogumelos Maravilha" < > cogumelosmaravilha@sapo.pt>: > > From cassandra.yaml: > hints_directory: /mnt/cassandra/hints > data_file_directories: > - /mnt/cassandra/data > commitlog_directory: /mnt/cassandra/commitlog > saved_caches_directory: /mnt/cassandra/saved_caches > > drwxr-xr-x 3 cassandra cassandra 23 Apr 5 16:03 mnt/ > > drwxr-xr-x 6 cassandra cassandra 68 Apr 5 16:17 ./ > drwxr-xr-x 3 cassandra cassandra 23 Apr 5 16:03 ../ > drwxr-xr-x 2 cassandra cassandra 80 Apr 6 10:07 commitlog/ > drwxr-xr-x 8 cassandra cassandra 124 Apr 5 16:17 data/ > drwxr-xr-x 2 cassandra cassandra 72 Apr 5 16:20 hints/ > drwxr-xr-x 2 cassandra cassandra 49 Apr 5 20:17 saved_caches/ > > cassand+ 2267 1 99 10:18 ? 00:02:56 java > -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities > -XX:Threa... > > /dev/mapper/um_vg-xfs_lv 885G 27G 858G 4% /mnt > > On /etc/security/limits.conf > > * - memlock unlimited > * - nofile 100000 > * - nproc 32768 > * - as unlimited > > On /etc/security/limits.d/cassandra.conf > > cassandra - memlock unlimited > cassandra - nofile 100000 > cassandra - as unlimited > cassandra - nproc 32768 > > On /etc/sysctl.conf > > vm.max_map_count = 1048575 > > On /etc/systcl.d/cassanda.conf > > vm.max_map_count = 1048575 > net.ipv4.tcp_keepalive_time=600 > On /etc/pam.d/su > ... > session required pam_limits.so > ... > > Distro is the currently Ubuntu LTS. > Thanks > > > > On 04/06/2017 10:39 AM, benjamin roth wrote: > > Cassandra cannot write an SSTable to disk. Are you sure the disk/volume > where SSTables reside (normally /var/lib/cassandra/data) is writeable for > the CS user and has enough free space? > The CDC warning also implies that. > The other warnings indicate you are probably not running CS as root and > you did not set an appropriate limit for max open files. Running out of > open files can also be a reason for the IO error. > > 2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha >: > >> Hi list, >> >> I'm using C* 3.10 in a 6 nodes cluster RF=2. All instances type >> i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G. I have >> one node that is always dieing and I don't understand why. Can anyone >> give me some hints please. All nodes using the same configuration. >> >> Thanks in advance. >> >> INFO [IndexSummaryManager:1] 2017-04-06 05:22:18,352 >> IndexSummaryRedistribution.java:75 - Redistributing index summaries >> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800 >> CassandraDaemon.java:229 - Exception in thread >> Thread[MemtablePostFlush:22,5,main] >> org.apache.cassandra.io.FSWriteError: java.io.IOException: Input/output >> error >> at >> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyIn >> ternal(SequentialWriter.java:173) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.util.SequentialWriter.syncInternal(S >> equentialWriter.java:185) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.compress.CompressedSequentialWriter. >> access$100(CompressedSequentialWriter.java:38) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.compress.CompressedSequentialWriter$ >> TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.utils.concurrent.Transactional$Abstract >> Transactional.prepareToCommit(Transactional.java:173) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.util.SequentialWriter.prepareToCommi >> t(SequentialWriter.java:358) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.sstable.format.big.BigTableWriter$Tr >> ansactionalProxy.doPrepare(BigTableWriter.java:367) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.utils.concurrent.Transactional$Abstract >> Transactional.prepareToCommit(Transactional.java:173) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.sstable.format.SSTableWriter.prepare >> ToCommit(SSTableWriter.java:281) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.pre >> pareToCommit(SimpleSSTableMultiWriter.java:101) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtabl >> e(ColumnFamilyStore.java:1153) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFa >> milyStore.java:1086) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> ~[na:1.8.0_121] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> [na:1.8.0_121] >> at >> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$th >> readLocalDeallocator$0(NamedThreadFactory.java:79) >> [apache-cassandra-3.10.jar:3.10] >> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121] >> Caused by: java.io.IOException: Input/output error >> at sun.nio.ch.FileDispatcherImpl.force0(Native Method) >> ~[na:1.8.0_121] >> at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76) >> ~[na:1.8.0_121] >> at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388) >> ~[na:1.8.0_121] >> at org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyIn >> ternal(SequentialWriter.java:169) >> ~[apache-cassandra-3.10.jar:3.10] >> ... 15 common frames omitted >> INFO [IndexSummaryManager:1] 2017-04-06 06:22:18,366 >> IndexSummaryRedistribution.java:75 - Redistributing index summaries >> ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525 >> CassandraDaemon.java:229 - Exception in thread >> Thread[MemtablePostFlush:31,5,main] >> org.apache.cassandra.io.FSWriteError: java.io.IOException: Input/output >> error >> at >> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyIn >> ternal(SequentialWriter.java:173) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.util.SequentialWriter.syncInternal(S >> equentialWriter.java:185) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.compress.CompressedSequentialWriter. >> access$100(CompressedSequentialWriter.java:38) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.compress.CompressedSequentialWriter$ >> TransactionalProxy.doPrepare(CompressedSequentialWriter.java:307) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.utils.concurrent.Transactional$Abstract >> Transactional.prepareToCommit(Transactional.java:173) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.util.SequentialWriter.prepareToCommi >> t(SequentialWriter.java:358) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.sstable.format.big.BigTableWriter$Tr >> ansactionalProxy.doPrepare(BigTableWriter.java:367) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.utils.concurrent.Transactional$Abstract >> Transactional.prepareToCommit(Transactional.java:173) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.sstable.format.SSTableWriter.prepare >> ToCommit(SSTableWriter.java:281) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.pre >> pareToCommit(SimpleSSTableMultiWriter.java:101) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtabl >> e(ColumnFamilyStore.java:1153) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFa >> milyStore.java:1086) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1142) >> ~[na:1.8.0_121] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:617) >> [na:1.8.0_121] >> at >> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$th >> readLocalDeallocator$0(NamedThreadFactory.java:79) >> [apache-cassandra-3.10.jar:3.10] >> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121] >> Caused by: java.io.IOException: Input/output error >> at sun.nio.ch.FileDispatcherImpl.force0(Native Method) >> ~[na:1.8.0_121] >> at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76) >> ~[na:1.8.0_121] >> at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388) >> ~[na:1.8.0_121] >> at org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158) >> ~[apache-cassandra-3.10.jar:3.10] >> at >> org.apache.cassandra.io.util.SequentialWriter.syncDataOnlyIn >> ternal(SequentialWriter.java:169) >> ~[apache-cassandra-3.10.jar:3.10] >> ... 15 common frames omitted >> INFO [main] 2017-04-06 07:11:57,289 YamlConfigurationLoader.java:89 - >> Configuration location: file:/etc/cassandra/cassandra.yaml >> >> >> Some ERRORs messages: >> >> ERROR [MemtablePostFlush:2] 2017-04-05 23:35:46,339 >> CassandraDaemon.java:229 - Exception in thread >> Thread[MemtablePostFlush:2,5,main] >> ERROR [MemtablePostFlush:3] 2017-04-05 23:44:08,471 >> CassandraDaemon.java:229 - Exception in thread >> Thread[MemtablePostFlush:3,5,main] >> ERROR [MemtablePostFlush:4] 2017-04-05 23:54:41,224 >> CassandraDaemon.java:229 - Exception in thread >> Thread[MemtablePostFlush:4,5,main] >> ERROR [MessagingService-Incoming-/10.0.120.52] 2017-04-06 03:19:13,453 >> CassandraDaemon.java:229 - Exception in thread >> Thread[MessagingService-Incoming-/10.0.120.52,5,main] >> ERROR [epollEventLoopGroup-2-6] 2017-04-06 03:24:41,006 >> CassandraDaemon.java:229 - Exception in thread >> Thread[epollEventLoopGroup-2-6,10,main] >> ERROR [Native-Transport-Requests-36] 2017-04-06 03:25:45,915 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [Native-Transport-Requests-49] 2017-04-06 03:25:45,915 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [IndexSummaryManager:1] 2017-04-06 03:25:45,915 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [Native-Transport-Requests-69] 2017-04-06 03:25:45,916 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [Native-Transport-Requests-46] 2017-04-06 03:26:18,465 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [SharedPool-Worker-136] 2017-04-06 03:26:18,465 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [Native-Transport-Requests-156] 2017-04-06 03:26:18,465 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [SharedPool-Worker-92] 2017-04-06 03:26:24,696 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [Native-Transport-Requests-48] 2017-04-06 03:26:24,696 ?:? - JVM >> state determined to be unstable. Exiting forcefully due to: >> ERROR [Native-Transport-Requests-66] 2017-04-06 03:26:55,808 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [Native-Transport-Requests-77] 2017-04-06 03:26:55,808 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [GossipTasks:1] 2017-04-06 03:26:55,808 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [Native-Transport-Requests-133] 2017-04-06 03:26:55,808 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [Native-Transport-Requests-135] 2017-04-06 03:26:55,808 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [ScheduledFastTasks:1] 2017-04-06 03:26:55,808 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [Native-Transport-Requests-70] 2017-04-06 03:27:11,569 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [IndexSummaryManager:1] 2017-04-06 03:27:17,821 >> CassandraDaemon.java:229 - Exception in thread >> Thread[IndexSummaryManager:1,1,main] >> ERROR [Native-Transport-Requests-103] 2017-04-06 03:27:24,049 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [Native-Transport-Requests-69] 2017-04-06 03:27:24,049 >> SEPWorker.java:145 - Failed to execute task, unexpected exception killed >> worker: {} >> ERROR [SharedPool-Worker-98] 2017-04-06 03:27:24,049 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [MessagingService-Incoming-/10.0.120.52] 2017-04-06 03:27:55,079 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [epollEventLoopGroup-2-5] 2017-04-06 03:27:55,079 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: >> ERROR [Native-Transport-Requests-64] 2017-04-06 03:28:43,285 >> SEPWorker.java:145 - Failed to execute task, unexpected exception killed >> worker: {} >> ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800 >> CassandraDaemon.java:229 - Exception in thread >> Thread[MemtablePostFlush:22,5,main] >> ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525 >> CassandraDaemon.java:229 - Exception in thread >> Thread[MemtablePostFlush:31,5,main] >> >> Also some WARNs: >> >> WARN [main] 2017-04-06 09:26:49,725 CLibrary.java:178 - Unable to lock >> JVM memory (ENOMEM). This can result in part of the JVM being swapped >> out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run >> Cassandra as root. >> >> WARN [main] 2017-04-06 09:25:07,355 StartupChecks.java:157 - JMX is not >> enabled to receive remote connections. Please see cassandra-env.sh for >> more info. >> >> WARN [main] 2017-04-06 09:25:07,369 SigarLibrary.java:174 - Cassandra >> server running in degraded mode. Is swap disabled? : true, Address >> space adequate? : true, nofile limit adequate? : false, nproc limit >> adequate? : true >> >> WARN [main] 2017-04-06 09:25:07,091 DatabaseDescriptor.java:493 - Small >> cdc volume detected at /var/lib/cassandra/cdc_raw; setting >> cdc_total_space_in_mb to 2502. You can override this in cassandra.yaml >> >> >> > > > > --001a113e20b6748ceb054cce6589 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Notice
.SimpleSeedProvider{seeds=3D10.10= 0.100.19, 10.100.100.85, 10.100.100.185, 10.100.100.161, 10.100.100.52, 10.100.1000.213};

Why do you have all six of your nodes = as seeds? is it possible that the last one you added used itself as the see= d and is isolated?

On Thu, Apr 6, 2017 at 6:48 AM, Cogumelos Maravilha <c= ogumelosmaravilha@sapo.pt> wrote:
=20 =20 =20

Yes C* is running as cassandra:

cassand+=C2=A0 2267=C2=A0=C2=A0=C2=A0=C2=A0 1 99 10:18 ?=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 00:02:56 java -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...

INFO=C2=A0 [main] 2017-04-06 10:35:42,956 Config.java:474 - N= ode configuration:[allocate_tokens_for_keyspace=3Dnull; authenticator=3DPasswordAuthenticator; authorizer=3DCassandraAuthorizer; auto_bootstrap=3Dtrue; auto_snapshot=3Dtrue; back_pressure_enabled=3Dfalse; back_pressure_strategy=3Dorg.a= pache.cassandra.net.RateBasedBackPressure{high_ratio=3D0.9, factor=3D5, flow=3DFAST}; batch_size_fail_threshold_in_kb=3D50; batch_size_warn_threshold_in_kb=3D5; batchlog_replay_throttle_in_kb=3D1024; broadcast_address=3Dnull; broadcast_rpc_address=3Dnull; buffer_pool_use_heap_if_exhausted=3Dtrue; cas_contention_timeout_in_ms=3D6000000; cdc_enabled=3Dfalse; cdc_free_space_check_interval_ms=3D250; cdc_raw_directory=3Dnull= ; cdc_total_space_in_mb=3D0; client_encryption_options=3D<REDACTED>; cluster_name=3Dcom= pany; column_index_cache_size_in_kb=3D2; column_index_size_in_kb=3D64; commit_failure_policy=3Dignore; commitlog_compression=3Dnull; commitlog_directory=3D/mnt/cassandra/commitlog; commitlog_max_compression_buffers_in_pool=3D3; commitlog_periodic_queue_size=3D-1; commitlog_segment_size_in_mb= =3D32; commitlog_sync=3Dperiodic; commitlog_sync_batch_window_in_ms=3DN= aN; commitlog_sync_period_in_ms=3D10000; commitlog_total_space_in_mb=3Dnull; compaction_large_partition_warning_threshold_mb=3D100; compaction_throughput_mb_per_sec=3D16; concurrent_compactors=3Dn= ull; concurrent_counter_writes=3D32; concurrent_materialized_view_writes=3D32; concurrent_reads=3D32; concurrent_replicates=3Dnull; concurrent_writes=3D32; counter_cache_keys_to_save=3D2147483647; counter_cache_save_period=3D7200; counter_cache_size_in_mb=3Dnul= l; counter_write_request_timeout_in_ms=3D6000000; credentials_cache_max_entries=3D1000; credentials_update_interval_in_ms=3D-1; credentials_validity_in_ms=3D2000; cross_node_timeout=3Dfalse; data_file_directories=3D[Ljava.lang.String;@223f3642; disk_access_mode=3Dauto; disk_failure_policy=3Dignore; disk_optimization_estimate_percentile=3D0.95; disk_optimization_page_cross_chance=3D0.1; disk_optimization_strategy=3Dssd; dynamic_snitch=3Dtrue; dynamic_snitch_badness_threshold=3D0.1; dynamic_snitch_reset_interval_in_ms=3D600000; dynamic_snitch_update_interval_in_ms=3D100; enable_scripted_user_defined_functions=3Dfalse; enable_user_defined_functions=3Dfalse; enable_user_defined_functions_threads=3Dtrue; encryption_options=3Dnull; endpoint_snitch=3DSimpleSnitch; file_cache_size_in_mb=3Dnull; gc_log_threshold_in_ms=3D200; gc_warn_threshold_in_ms=3D1000; hinted_handoff_disabled_datacenters=3D[]; hinted_handoff_enabled=3Dtrue; hinted_handoff_throttle_in_kb=3D1= 024; hints_compression=3Dnull; hints_directory=3D/mnt/cassandra/hints= ; hints_flush_period_in_ms=3D10000; incremental_backups=3Dfalse; index_interval=3Dnull; index_summary_capacity_in_mb=3Dnull; index_summary_resize_interval_in_minutes=3D60; initial_token=3Dn= ull; inter_dc_stream_throughput_outbound_megabits_per_sec=3D200; inter_dc_tcp_nodelay=3Dfalse; internode_authenticator=3Dnull; internode_compression=3Ddc; internode_recv_buff_size_in_bytes=3D= 0; internode_send_buff_size_in_bytes=3D0; key_cache_keys_to_save=3D2147483647; key_cache_save_period=3D144= 00; key_cache_size_in_mb=3Dnull; listen_address=3D10.100.100.213; listen_interface=3Dnull; listen_interface_prefer_ipv6=3Dfalse; listen_on_broadcast_address=3Dfalse; max_hint_window_in_ms=3D10800000; max_hints_delivery_threads=3D2; max_hints_file_size_in_mb=3D128; max_mutation_size_in_kb=3Dnull; max_streaming_retries=3D3; max_value_size_in_mb=3D256; memtable_allocation_type=3Dheap_buff= ers; memtable_cleanup_threshold=3Dnull; memtable_flush_writers=3D0; memtable_heap_space_in_mb=3Dnull; memtable_offheap_space_in_mb= =3Dnull; min_free_space_per_drive_in_mb=3D50; native_transport_max_concurrent_connections=3D-1; native_transport_max_concurrent_connections_per_ip=3D-1; native_transport_max_frame_size_in_mb=3D256; native_transport_max_threads=3D128; native_transport_port=3D9042= ; native_transport_port_ssl=3Dnull; num_tokens=3D256; otc_coalescing_strategy=3DTIMEHORIZON; otc_coalescing_window_us= =3D200; partitioner=3Dorg.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=3D1000; permissions_update_interval_in_ms=3D-1; permissions_validity_in_ms=3D2000; phi_convict_threshold=3D8.0; prepared_statements_cache_size_mb=3Dnull; range_request_timeout_in_ms=3D6000000; read_request_timeout_in_ms=3D6000000; request_scheduler=3Dorg.apache.cassandra.scheduler.NoSchedu= ler; request_scheduler_id=3Dnull; request_scheduler_options=3Dnull; request_timeout_in_ms=3D6000000; role_manager=3DCassandraRoleMan= ager; roles_cache_max_entries=3D1000; roles_update_interval_in_ms=3D-1= ; roles_validity_in_ms=3D2000; row_cache_class_name=3Dorg.apache.cassandra.cache.OHCProvid= er; row_cache_keys_to_save=3D2147483647; row_cache_save_period=3D0; row_cache_size_in_mb=3D0; rpc_address=3D10.100.100.213; rpc_interface=3Dnull; rpc_interface_prefer_ipv6=3Dfalse; rpc_keepalive=3Dtrue; rpc_listen_backlog=3D50; rpc_max_threads=3D2147483647; rpc_min_threads=3D16; rpc_port=3D9160; rpc_recv_buff_size_in_bytes=3Dnull; rpc_send_buff_size_in_bytes=3Dnull; rpc_server_type=3Dsync; saved_caches_directory=3D/mnt/cassandra/saved_caches; seed_provider=3Dorg.apache.cassandra.locator.SimpleSeedProvider{s= eeds=3D10.100.100.19, 10.100.100.85, 10.100.100.185, 10.100.100.161, 10.100.100.52, 10.100.1000.213}; server_encryption_options=3D<REDACTED>; slow_query_log_timeout_in_ms=3D6000000; snapshot_before_compaction=3Dfalse; ssl_storage_port=3D7001; sstable_preemptive_open_interval_in_mb=3D50; start_native_transport=3Dtrue; start_rpc=3Dfalse; storage_port=3D7000= ; stream_throughput_outbound_megabits_per_sec=3D200; streaming_keep_alive_period_in_secs=3D300; streaming_socket_timeout_in_ms=3D86400000; thrift_framed_transport_size_in_mb=3D15; thrift_max_message_length_in_mb=3D16; thrift_prepared_statements_cache_size_mb=3Dnull; tombstone_failure_threshold=3D100000; tombstone_warn_threshold= =3D1000; tracetype_query_ttl=3D86400; tracetype_repair_ttl=3D604800; transparent_data_encryption_options=3Dorg.apache.cassandra.config= .TransparentDataEncryptionOptions@38c5cc4c; trickle_fsync=3Dfalse; trickle_fsync_interval_in_kb=3D10240; truncate_request_timeout_in_ms=3D6000000; unlogged_batch_across_partitions_warn_threshold=3D10; user_defined_function_fail_timeout=3D1500; user_defined_function_warn_timeout=3D500; user_function_timeout_policy=3Ddie; windows_timer_interval=3D1; write_request_timeout_in_ms=3D6000000]

Thanks


On 04/06/2017 11:30= AM, benjamin roth wrote:
Have you checked the effective limits of a running CS process?
Is CS run as Cassandra? Just to rule out missing file perms.=C2=A0


Am 06.04.2017 12:24 schrieb "Cogumelos Maravilha" <cogumelosmaravilha@sapo.pt>:

From cassandra.yaml:

hints_directory: /mnt/cassandra/hints
data_file_directories:
=C2=A0=C2=A0=C2=A0 - /mnt/cassandra/data
commitlog_directory: /mnt/cassandra/commitlog
saved_caches_directory: /mnt/cassandra/saved_caches

drwxr-xr-x=C2=A0=C2=A0 3 cassandra cassandra=C2=A0=C2=A0= 23 Apr=C2=A0 5 16:03 mnt/

drwxr-xr-x 6 cassandra cassandra=C2=A0 68 Apr=C2=A0 5 16= :17 ./
drwxr-xr-x 3 cassandra cassandra=C2=A0 23 Apr=C2=A0 5 16:= 03 ../
drwxr-xr-x 2 cassandra cassandra=C2=A0 80 Apr=C2=A0 6 10:= 07 commitlog/
drwxr-xr-x 8 cassandra cassandra 124 Apr=C2=A0 5 16:17 data/
drwxr-xr-x 2 cassandra cassandra=C2=A0 72 Apr=C2=A0 5 16:= 20 hints/
drwxr-xr-x 2 cassandra cassandra=C2=A0 49 Apr=C2=A0 5 20:= 17 saved_caches/

cassand+=C2=A0 2267=C2=A0=C2=A0=C2=A0=C2=A0 1 99 10:18 ?= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 00:02:56 java -Xloggc:/var/log/cassandra/gc.log -ea -XX:+UseThreadPriorities -XX:Threa...

/dev/mapper/um_vg-xfs_lv=C2=A0 885G=C2=A0=C2=A0 27G=C2= =A0 858G=C2=A0=C2=A0 4% /mnt

On /etc/security/limits.conf

* =C2=A0=C2=A0 =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 -=C2=A0 = =C2=A0=C2=A0=C2=A0 =C2=A0memlock =C2=A0=C2=A0=C2=A0 =C2=A0unlimited
* =C2=A0 =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 - =C2=A0= =C2=A0=C2=A0 =C2=A0nofile =C2=A0=C2=A0=C2=A0 =C2=A0100000
* =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0- =C2= =A0=C2=A0=C2=A0 =C2=A0nproc =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0327= 68
*=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 - =C2= =A0=C2=A0=C2=A0 =C2=A0as =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0= unlimited

On /etc/security/limits.d/cassandra.conf

cassandra=C2=A0 -=C2=A0 memlock=C2=A0 unlimited
cassandra=C2=A0 -=C2=A0 nofile=C2=A0=C2=A0 100000
cassandra=C2=A0 -=C2=A0 as=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 unlimited
cassandra=C2=A0 -=C2=A0 nproc=C2=A0=C2=A0=C2=A0 32768

On /etc/sysctl.conf

vm.max_map_count =3D 1048575

On /etc/systcl.d/cassanda.conf

vm.max_map_count =3D 1048575
net.ipv4.tcp_keepalive_time=3D600

On /etc/pam.d/su
...
session=C2=A0=C2=A0=C2=A0 required=C2=A0=C2=A0 pam_limits.s= o
...

Distro is the currently Ubuntu LTS.
Thanks



On 04/06/2017 10:39 AM, benjamin roth wrote:
Cassandra cannot write an SSTable to disk. Are you sure the disk/volume where SSTables reside (normally /var/lib/cassandra/data) is writeable for the CS user and has enough free space?
The CDC warning also implies that.
The other warnings indicate you are probably not running CS as root and you did not set an appropriate limit for max open files. Running out of open files can also be a reason for the IO error.

2017-04-06 11:34 GMT+02:00 Cogumelos Maravilha <= ;cogumelosm= aravilha@sapo.pt>:
Hi list,

I'm using C* 3.10 in a 6 nodes cluster RF=3D2= . All instances type
i3.xlarge (AWS) with 32GB, 2 cores and SSD LVM XFS formated 885G. I have
one node that is always dieing and I don't understand why. Can anyone
give me some hints please. All nodes using the same configuration.

Thanks in advance.

INFO=C2=A0 [IndexSummaryManager:1] 2017-04-06 05:22:18,352
IndexSummaryRedistribution.java:75 - Redistributing index summaries
ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
CassandraDaemon.java:229 - Exception in thread Thread[MemtablePostFlush:22,5,main]
org.apache.cassandra.io.FSWriteError: java.io.IOException: Input/output
error
=C2=A0 =C2=A0 at
org.apache.cassandra.io.util.SequentialWrite= r.syncDataOnlyInternal(SequentialWriter.java:173)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.util.SequentialWrite= r.syncInternal(SequentialWriter.java:185)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.compress.CompressedSequentialW= riter.access$100(CompressedSequentialWriter.java:38)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.compress.CompressedSequentialW= riter$TransactionalProxy.doPrepare(CompressedSequentialWriter.jav= a:307)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.utils.concurrent.Transa= ctional$AbstractTransactional.prepareToCommit(Transactional.java:= 173)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.util.SequentialWrite= r.prepareToCommit(SequentialWriter.java:358)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.sstable.format.big.BigTableWri= ter$TransactionalProxy.doPrepare(BigTableWriter.java:367)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.utils.concurrent.Transa= ctional$AbstractTransactional.prepareToCommit(Transactional.java:= 173)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.sstable.format.SSTableWriter.p= repareToCommit(SSTableWriter.java:281)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.sstable.SimpleSSTableMultiWrit= er.prepareToCommit(SimpleSSTableMultiWriter.java:101)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.db.ColumnFamilyStore$Fl= ush.flushMemtable(ColumnFamilyStore.java:1153)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.db.ColumnFamilyStore$Fl= ush.run(ColumnFamilyStore.java:1086)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
java.util.concurrent.ThreadPoolExecutor.runW= orker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_121]
=C2=A0 =C2=A0 at
java.util.concurrent.ThreadPoolExecutor$Work= er.run(ThreadPoolExecutor.java:617)
[na:1.8.0_121]
=C2=A0 =C2=A0 at
org.apache.cassandra.concurrent.NamedThreadF= actory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79= )
[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
Caused by: java.io.IOException: Input/output error
=C2=A0 =C2=A0 at sun.nio.ch.FileDispatcherImpl.force0(Native Method) ~[na:1.8.0_121]
=C2=A0 =C2=A0 at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
~[na:1.8.0_121]
=C2=A0 =C2=A0 at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388)
~[na:1.8.0_121]
=C2=A0 =C2=A0 at org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.util.SequentialWrite= r.syncDataOnlyInternal(SequentialWriter.java:169)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 ... 15 common frames omitted
INFO=C2=A0 [IndexSummaryManager:1] 2017-04-06 06:22:18,366
IndexSummaryRedistribution.java:75 - Redistributing index summaries
ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525
CassandraDaemon.java:229 - Exception in thread Thread[MemtablePostFlush:31,5,main]
org.apache.cassandra.io.FSWriteError: java.io.IOException: Input/output
error
=C2=A0 =C2=A0 at
org.apache.cassandra.io.util.SequentialWrite= r.syncDataOnlyInternal(SequentialWriter.java:173)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.util.SequentialWrite= r.syncInternal(SequentialWriter.java:185)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.compress.CompressedSequentialW= riter.access$100(CompressedSequentialWriter.java:38)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.compress.CompressedSequentialW= riter$TransactionalProxy.doPrepare(CompressedSequentialWriter.jav= a:307)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.utils.concurrent.Transa= ctional$AbstractTransactional.prepareToCommit(Transactional.java:= 173)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.util.SequentialWrite= r.prepareToCommit(SequentialWriter.java:358)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.sstable.format.big.BigTableWri= ter$TransactionalProxy.doPrepare(BigTableWriter.java:367)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.utils.concurrent.Transa= ctional$AbstractTransactional.prepareToCommit(Transactional.java:= 173)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.sstable.format.SSTableWriter.p= repareToCommit(SSTableWriter.java:281)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.sstable.SimpleSSTableMultiWrit= er.prepareToCommit(SimpleSSTableMultiWriter.java:101)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.db.ColumnFamilyStore$Fl= ush.flushMemtable(ColumnFamilyStore.java:1153)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.db.ColumnFamilyStore$Fl= ush.run(ColumnFamilyStore.java:1086)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
java.util.concurrent.ThreadPoolExecutor.runW= orker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_121]
=C2=A0 =C2=A0 at
java.util.concurrent.ThreadPoolExecutor$Work= er.run(ThreadPoolExecutor.java:617)
[na:1.8.0_121]
=C2=A0 =C2=A0 at
org.apache.cassandra.concurrent.NamedThreadF= actory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79= )
[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
Caused by: java.io.IOException: Input/output error
=C2=A0 =C2=A0 at sun.nio.ch.FileDispatcherImpl.force0(Native Method) ~[na:1.8.0_121]
=C2=A0 =C2=A0 at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
~[na:1.8.0_121]
=C2=A0 =C2=A0 at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388)
~[na:1.8.0_121]
=C2=A0 =C2=A0 at org.apache.cassandra.utils.SyncUtil.force(SyncUtil.java:158)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 at
org.apache.cassandra.io.util.SequentialWrite= r.syncDataOnlyInternal(SequentialWriter.java:169)
~[apache-cassandra-3.10.jar:3.10]
=C2=A0 =C2=A0 ... 15 common frames omitted
INFO=C2=A0 [main] 2017-04-06 07:11:57,289 YamlConfigurationLoader.java:89 -
Configuration location: file:/etc/cassandra/ca= ssandra.yaml


Some ERRORs messages:

ERROR [MemtablePostFlush:2] 2017-04-05 23:35:46,339
CassandraDaemon.java:229 - Exception in thread Thread[MemtablePostFlush:2,5,main]
ERROR [MemtablePostFlush:3] 2017-04-05 23:44:08,471
CassandraDaemon.java:229 - Exception in thread Thread[MemtablePostFlush:3,5,main]
ERROR [MemtablePostFlush:4] 2017-04-05 23:54:41,224
CassandraDaemon.java:229 - Exception in thread Thread[MemtablePostFlush:4,5,main]
ERROR [MessagingService-Incoming-/10.0.120.52] 2017-04-06 03:19:13,453
CassandraDaemon.java:229 - Exception in thread Thread[MessagingService-Incoming-/10.0.120.52,5= ,main]
ERROR [epollEventLoopGroup-2-6] 2017-04-06 03:24:41,006
CassandraDaemon.java:229 - Exception in thread Thread[epollEventLoopGroup-2-6,10,main]
ERROR [Native-Transport-Requests-36] 2017-04-06 03:25:45,915
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [Native-Transport-Requests-49] 2017-04-06 03:25:45,915
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [IndexSummaryManager:1] 2017-04-06 03:25:45,915
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [Native-Transport-Requests-69] 2017-04-06 03:25:45,916
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [Native-Transport-Requests-46] 2017-04-06 03:26:18,465
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [SharedPool-Worker-136] 2017-04-06 03:26:18,465
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [Native-Transport-Requests-156] 2017-04-06 03:26:18,465
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [SharedPool-Worker-92] 2017-04-06 03:26:24,696
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [Native-Transport-Requests-48] 2017-04-06 03:26:24,696 ?:? - JVM
state determined to be unstable.=C2=A0 Exiting forcefully due to:
ERROR [Native-Transport-Requests-66] 2017-04-06 03:26:55,808
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [Native-Transport-Requests-77] 2017-04-06 03:26:55,808
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [GossipTasks:1] 2017-04-06 03:26:55,808
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [Native-Transport-Requests-133] 2017-04-06 03:26:55,808
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [Native-Transport-Requests-135] 2017-04-06 03:26:55,808
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [ScheduledFastTasks:1] 2017-04-06 03:26:55,808
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [Native-Transport-Requests-70] 2017-04-06 03:27:11,569
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [IndexSummaryManager:1] 2017-04-06 03:27:17,821
CassandraDaemon.java:229 - Exception in thread Thread[IndexSummaryManager:1,1,main]
ERROR [Native-Transport-Requests-103] 2017-04-06 03:27:24,049
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [Native-Transport-Requests-69] 2017-04-06 03:27:24,049
SEPWorker.java:145 - Failed to execute task, unexpected exception killed
worker: {}
ERROR [SharedPool-Worker-98] 2017-04-06 03:27:24,049
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [MessagingService-Incoming-/10.0.120.52] 2017-04-06 03:27:55,079
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [epollEventLoopGroup-2-5] 2017-04-06 03:27:55,079
JVMStabilityInspector.java:142 - JVM state determined to be unstable.
Exiting forcefully due to:
ERROR [Native-Transport-Requests-64] 2017-04-06 03:28:43,285
SEPWorker.java:145 - Failed to execute task, unexpected exception killed
worker: {}
ERROR [MemtablePostFlush:22] 2017-04-06 06:00:26,800
CassandraDaemon.java:229 - Exception in thread Thread[MemtablePostFlush:22,5,main]
ERROR [MemtablePostFlush:31] 2017-04-06 06:39:19,525
CassandraDaemon.java:229 - Exception in thread Thread[MemtablePostFlush:31,5,main]

Also some WARNs:

WARN=C2=A0 [main] 2017-04-06 09:26:49,725 CLibrary.java:178 - Unable to lock
JVM memory (ENOMEM). This can result in part of the JVM being swapped
out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run
Cassandra as root.

WARN=C2=A0 [main] 2017-04-06 09:25:07,355 StartupChecks.java:157 - JMX is not
enabled to receive remote connections. Please see cassandra-env.sh for
more info.

WARN=C2=A0 [main] 2017-04-06 09:25:07,369 SigarLibrary.java:174 - Cassandra
server running in degraded mode. Is swap disabled? : true,=C2=A0 Address
space adequate? : true,=C2=A0 nofile limit adequate? : false, nproc limit
adequate? : true

WARN=C2=A0 [main] 2017-04-06 09:25:07,091 DatabaseDescriptor.java:493 - Small
cdc volume detected at /var/lib/cassandra/cdc_raw; setting
cdc_total_space_in_mb to 2502.=C2=A0 You can override this in cassandra.yaml







--001a113e20b6748ceb054cce6589--