From commits-return-205238-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Wed Jan 10 21:34:28 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id F26CA18072F for ; Wed, 10 Jan 2018 21:34:27 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id E20D0160C40; Wed, 10 Jan 2018 20:34:27 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3325C160C1E for ; Wed, 10 Jan 2018 21:34:25 +0100 (CET) Received: (qmail 75955 invoked by uid 500); 10 Jan 2018 20:34:24 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 75943 invoked by uid 99); 10 Jan 2018 20:34:24 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Jan 2018 20:34:24 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 953F51A07E6 for ; Wed, 10 Jan 2018 20:34:23 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.661 X-Spam-Level: X-Spam-Status: No, score=-99.661 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LOTSOFHASH=0.25, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id qeJfNK6zQvHb for ; Wed, 10 Jan 2018 20:34:03 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 1773C5F620 for ; Wed, 10 Jan 2018 20:34:03 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 1673BE25E0 for ; Wed, 10 Jan 2018 20:34:02 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id BA926274E6 for ; Wed, 10 Jan 2018 20:34:00 +0000 (UTC) Date: Wed, 10 Jan 2018 20:34:00 +0000 (UTC) From: "Jeff Jirsa (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CASSANDRA-14161) node can't start, java.nio.file.AccessDeniedException to not existent file MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-14161?page=3Dcom.atl= assian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-14161: ----------------------------------- Description:=20 this exception cuts the node starting: {code} java.lang.RuntimeException: java.nio.file.AccessDeniedException: /var/lib/c= assandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_65= 79bf70-f625-11e7-8e56-71d63c551064.log {code} the offending file doesn't exists: {code} ls -la /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b3= 77/ total 160 drwxr-xr-x 3 cassandra cassandra 4096 ene 7 11:48 . drwxr-xr-x 26 cassandra cassandra 4096 dic 18 07:30 .. drwxr-xr-x 2 cassandra cassandra 4096 dic 18 07:30 backups -rw-r--r-- 1 cassandra cassandra 43 ene 7 03:50 mc-100-big-CompressionI= nfo.db -rw-r--r-- 1 cassandra cassandra 52 ene 7 03:50 mc-100-big-Data.db -rw-r--r-- 1 cassandra cassandra 8 ene 7 03:50 mc-100-big-Digest.crc32 -rw-r--r-- 1 cassandra cassandra 16 ene 7 03:50 mc-100-big-Filter.db -rw-r--r-- 1 cassandra cassandra 9 ene 7 03:50 mc-100-big-Index.db -rw-r--r-- 1 cassandra cassandra 4615 ene 7 03:50 mc-100-big-Statistics.d= b -rw-r--r-- 1 cassandra cassandra 59 ene 7 03:50 mc-100-big-Summary.db -rw-r--r-- 1 cassandra cassandra 92 ene 7 03:50 mc-100-big-TOC.txt -rw-r--r-- 1 cassandra cassandra 43 ene 7 11:48 mc-101-big-CompressionI= nfo.db -rw-r--r-- 1 cassandra cassandra 50 ene 7 11:48 mc-101-big-Data.db -rw-r--r-- 1 cassandra cassandra 10 ene 7 11:48 mc-101-big-Digest.crc32 -rw-r--r-- 1 cassandra cassandra 16 ene 7 11:48 mc-101-big-Filter.db -rw-r--r-- 1 cassandra cassandra 9 ene 7 11:48 mc-101-big-Index.db -rw-r--r-- 1 cassandra cassandra 4615 ene 7 11:48 mc-101-big-Statistics.d= b -rw-r--r-- 1 cassandra cassandra 59 ene 7 11:48 mc-101-big-Summary.db -rw-r--r-- 1 cassandra cassandra 92 ene 7 11:48 mc-101-big-TOC.txt -rw-r--r-- 1 cassandra cassandra 51 ene 6 17:50 mc-98-big-CompressionIn= fo.db -rw-r--r-- 1 cassandra cassandra 5226 ene 6 17:50 mc-98-big-Data.db -rw-r--r-- 1 cassandra cassandra 10 ene 6 17:50 mc-98-big-Digest.crc32 -rw-r--r-- 1 cassandra cassandra 16 ene 6 17:50 mc-98-big-Filter.db -rw-r--r-- 1 cassandra cassandra 9 ene 6 17:50 mc-98-big-Index.db -rw-r--r-- 1 cassandra cassandra 5654 ene 6 17:50 mc-98-big-Statistics.db -rw-r--r-- 1 cassandra cassandra 59 ene 6 17:50 mc-98-big-Summary.db -rw-r--r-- 1 cassandra cassandra 92 ene 6 17:50 mc-98-big-TOC.txt -rw-r--r-- 1 cassandra cassandra 43 ene 7 02:50 mc-99-big-CompressionIn= fo.db -rw-r--r-- 1 cassandra cassandra 50 ene 7 02:50 mc-99-big-Data.db -rw-r--r-- 1 cassandra cassandra 10 ene 7 02:50 mc-99-big-Digest.crc32 -rw-r--r-- 1 cassandra cassandra 16 ene 7 02:50 mc-99-big-Filter.db -rw-r--r-- 1 cassandra cassandra 9 ene 7 02:50 mc-99-big-Index.db -rw-r--r-- 1 cassandra cassandra 4615 ene 7 02:50 mc-99-big-Statistics.db -rw-r--r-- 1 cassandra cassandra 59 ene 7 02:50 mc-99-big-Summary.db -rw-r--r-- 1 cassandra cassandra 92 ene 7 02:50 mc-99-big-TOC.txt {code} permissions are ok, everything assigned to cassandra:cassandra the system.log: {code} INFO [main] 2018-01-10 10:43:29,367 YamlConfigurationLoader.java:89 - Conf= iguration location: file:/etc/cassandra/cassandra.yaml INFO [main] 2018-01-10 10:43:29,887 Config.java:481 - Node configuration:[= allocate_tokens_for_keyspace=3Dnull; authenticator=3DAllowAllAuthenticator;= authorizer=3DAllowAllAuthorizer; auto_bootstrap=3Dtrue; auto_snapshot=3Dtr= ue; back_pressure_enabled=3Dfalse; back_pressure_strategy=3Dnull; batch_siz= e_fail_threshold_in_kb=3D500; batch_size_warn_threshold_in_kb=3D50; batchlo= g_replay_throttle_in_kb=3D1024; broadcast_address=3Dnull; broadcast_rpc_add= ress=3Dnull; buffer_pool_use_heap_if_exhausted=3Dtrue; cas_contention_timeo= ut_in_ms=3D10000; cdc_enabled=3Dfalse; cdc_free_space_check_interval_ms=3D2= 50; cdc_raw_directory=3Dnull; cdc_total_space_in_mb=3D0; client_encryption_= options=3D; cluster_name=3DPAP Cluster; column_index_cache_size_i= n_kb=3D2; column_index_size_in_kb=3D64; commit_failure_policy=3Dstop; commi= tlog_compression=3Dnull; commitlog_directory=3D/var/lib/cassandra/commitlog= ; commitlog_max_compression_buffers_in_pool=3D3; commitlog_periodic_queue_s= ize=3D-1; commitlog_segment_size_in_mb=3D32; commitlog_sync=3Dperiodic; com= mitlog_sync_batch_window_in_ms=3DNaN; commitlog_sync_period_in_ms=3D10000; = commitlog_total_space_in_mb=3Dnull; compaction_large_partition_warning_thre= shold_mb=3D100; compaction_throughput_mb_per_sec=3D16; concurrent_compactor= s=3Dnull; concurrent_counter_writes=3D32; concurrent_materialized_view_writ= es=3D32; concurrent_reads=3D32; concurrent_replicates=3Dnull; concurrent_wr= ites=3D32; counter_cache_keys_to_save=3D2147483647; counter_cache_save_peri= od=3D7200; counter_cache_size_in_mb=3Dnull; counter_write_request_timeout_i= n_ms=3D50000; credentials_cache_max_entries=3D1000; credentials_update_inte= rval_in_ms=3D-1; credentials_validity_in_ms=3D2000; cross_node_timeout=3Dfa= lse; data_file_directories=3D[Ljava.lang.String;@79ca92b9; disk_access_mode= =3Dauto; disk_failure_policy=3Dstop; disk_optimization_estimate_percentile= =3D0.95; disk_optimization_page_cross_chance=3D0.1; disk_optimization_strat= egy=3Dssd; dynamic_snitch=3Dtrue; dynamic_snitch_badness_threshold=3D0.1; d= ynamic_snitch_reset_interval_in_ms=3D600000; dynamic_snitch_update_interval= _in_ms=3D100; enable_scripted_user_defined_functions=3Dfalse; enable_user_d= efined_functions=3Dfalse; enable_user_defined_functions_threads=3Dtrue; enc= ryption_options=3Dnull; endpoint_snitch=3DGossipingPropertyFileSnitch; file= _cache_size_in_mb=3Dnull; gc_log_threshold_in_ms=3D200; gc_warn_threshold_i= n_ms=3D1000; hinted_handoff_disabled_datacenters=3D[]; hinted_handoff_enabl= ed=3Dtrue; hinted_handoff_throttle_in_kb=3D1024; hints_compression=3Dnull; = hints_directory=3Dnull; hints_flush_period_in_ms=3D10000; incremental_backu= ps=3Dfalse; index_interval=3Dnull; index_summary_capacity_in_mb=3Dnull; ind= ex_summary_resize_interval_in_minutes=3D60; initial_token=3Dnull; inter_dc_= stream_throughput_outbound_megabits_per_sec=3D200; inter_dc_tcp_nodelay=3Df= alse; internode_authenticator=3Dnull; internode_compression=3Dall; internod= e_recv_buff_size_in_bytes=3D0; internode_send_buff_size_in_bytes=3D0; key_c= ache_keys_to_save=3D2147483647; key_cache_save_period=3D14400; key_cache_si= ze_in_mb=3Dnull; listen_address=3D192.121.166.40; listen_interface=3Dnull; = listen_interface_prefer_ipv6=3Dfalse; listen_on_broadcast_address=3Dfalse; = max_hint_window_in_ms=3D10800000; max_hints_delivery_threads=3D2; max_hints= _file_size_in_mb=3D128; max_mutation_size_in_kb=3Dnull; max_streaming_retri= es=3D3; max_value_size_in_mb=3D256; memtable_allocation_type=3Dheap_buffers= ; memtable_cleanup_threshold=3Dnull; memtable_flush_writers=3D0; memtable_h= eap_space_in_mb=3Dnull; memtable_offheap_space_in_mb=3Dnull; min_free_space= _per_drive_in_mb=3D50; native_transport_max_concurrent_connections=3D-1; na= tive_transport_max_concurrent_connections_per_ip=3D-1; native_transport_max= _frame_size_in_mb=3D256; native_transport_max_threads=3D128; native_transpo= rt_port=3D9042; native_transport_port_ssl=3Dnull; num_tokens=3D256; otc_bac= klog_expiration_interval_ms=3D200; otc_coalescing_enough_coalesced_messages= =3D8; otc_coalescing_strategy=3DDISABLED; otc_coalescing_window_us=3D200; p= artitioner=3Dorg.apache.cassandra.dht.Murmur3Partitioner; permissions_cache= _max_entries=3D1000; permissions_update_interval_in_ms=3D-1; permissions_va= lidity_in_ms=3D2000; phi_convict_threshold=3D8.0; prepared_statements_cache= _size_mb=3Dnull; range_request_timeout_in_ms=3D100000; read_request_timeout= _in_ms=3D100000; request_scheduler=3Dorg.apache.cassandra.scheduler.NoSched= uler; request_scheduler_id=3Dnull; request_scheduler_options=3Dnull; reques= t_timeout_in_ms=3D100000; role_manager=3DCassandraRoleManager; roles_cache_= max_entries=3D1000; roles_update_interval_in_ms=3D-1; roles_validity_in_ms= =3D2000; row_cache_class_name=3Dorg.apache.cassandra.cache.OHCProvider; row= _cache_keys_to_save=3D2147483647; row_cache_save_period=3D0; row_cache_size= _in_mb=3D0; rpc_address=3Dlocalhost; rpc_interface=3Dnull; rpc_interface_pr= efer_ipv6=3Dfalse; rpc_keepalive=3Dtrue; rpc_listen_backlog=3D50; rpc_max_t= hreads=3D2147483647; rpc_min_threads=3D16; rpc_port=3D9160; rpc_recv_buff_s= ize_in_bytes=3Dnull; rpc_send_buff_size_in_bytes=3Dnull; rpc_server_type=3D= sync; saved_caches_directory=3D/var/lib/cassandra/saved_caches; seed_provid= er=3Dorg.apache.cassandra.locator.SimpleSeedProvider{seeds=3D63.142.255.107= ,37.247.116.68}; server_encryption_options=3D; slow_query_log_tim= eout_in_ms=3D500; snapshot_before_compaction=3Dfalse; ssl_storage_port=3D70= 71; sstable_preemptive_open_interval_in_mb=3D50; start_native_transport=3Dt= rue; start_rpc=3Dfalse; storage_port=3D7070; stream_throughput_outbound_meg= abits_per_sec=3D200; streaming_keep_alive_period_in_secs=3D300; streaming_s= ocket_timeout_in_ms=3D86400000; thrift_framed_transport_size_in_mb=3D15; th= rift_max_message_length_in_mb=3D16; thrift_prepared_statements_cache_size_m= b=3Dnull; tombstone_failure_threshold=3D100000; tombstone_warn_threshold=3D= 1000; tracetype_query_ttl=3D86400; tracetype_repair_ttl=3D604800; transpare= nt_data_encryption_options=3Dorg.apache.cassandra.config.TransparentDataEnc= ryptionOptions@1460a8c0; trickle_fsync=3Dfalse; trickle_fsync_interval_in_k= b=3D10240; truncate_request_timeout_in_ms=3D600000; unlogged_batch_across_p= artitions_warn_threshold=3D10; user_defined_function_fail_timeout=3D1500; u= ser_defined_function_warn_timeout=3D500; user_function_timeout_policy=3Ddie= ; windows_timer_interval=3D1; write_request_timeout_in_ms=3D50000] INFO [main] 2018-01-10 10:43:29,888 DatabaseDescriptor.java:367 - DiskAcce= ssMode 'auto' determined to be mmap, indexAccessMode is mmap INFO [main] 2018-01-10 10:43:29,888 DatabaseDescriptor.java:421 - Global m= emtable on-heap threshold is enabled at 374MB INFO [main] 2018-01-10 10:43:29,888 DatabaseDescriptor.java:425 - Global m= emtable off-heap threshold is enabled at 374MB WARN [main] 2018-01-10 10:43:29,897 DatabaseDescriptor.java:470 - Small co= mmitlog volume detected at /var/lib/cassandra/commitlog; setting commitlog_= total_space_in_mb to 7680. You can override this in cassandra.yaml WARN [main] 2018-01-10 10:43:29,899 DatabaseDescriptor.java:497 - Small cd= c volume detected at /var/lib/cassandra/cdc_raw; setting cdc_total_space_in= _mb to 3840. You can override this in cassandra.yaml WARN [main] 2018-01-10 10:43:30,022 DatabaseDescriptor.java:546 - Only 18.= 837GiB free across all data volumes. Consider adding more capacity to your = cluster or removing obsolete snapshots INFO [main] 2018-01-10 10:43:30,066 RateBasedBackPressure.java:123 - Initi= alized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window si= ze: 50000. INFO [main] 2018-01-10 10:43:30,066 DatabaseDescriptor.java:725 - Back-pre= ssure is disabled with strategy null. INFO [main] 2018-01-10 10:43:30,250 GossipingPropertyFileSnitch.java:64 - = Loaded cassandra-topology.properties for compatibility INFO [main] 2018-01-10 10:43:30,329 JMXServerUtils.java:249 - Configured J= MX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi INFO [main] 2018-01-10 10:43:30,335 CassandraDaemon.java:471 - Hostname: u= k1.plugandplay.click INFO [main] 2018-01-10 10:43:30,335 CassandraDaemon.java:478 - JVM vendor/= version: Java HotSpot(TM) 64-Bit Server VM/1.8.0_131 INFO [main] 2018-01-10 10:43:30,336 CassandraDaemon.java:479 - Heap size: = 1.463GiB/1.463GiB INFO [main] 2018-01-10 10:43:30,336 CassandraDaemon.java:484 - Code Cache = Non-heap memory: init =3D 2555904(2496K) used =3D 4545344(4438K) committed = =3D 4587520(4480K) max =3D 251658240(245760K) INFO [main] 2018-01-10 10:43:30,337 CassandraDaemon.java:484 - Metaspace N= on-heap memory: init =3D 0(0K) used =3D 19068664(18621K) committed =3D 1979= 1872(19328K) max =3D -1(-1K) INFO [main] 2018-01-10 10:43:30,337 CassandraDaemon.java:484 - Compressed = Class Space Non-heap memory: init =3D 0(0K) used =3D 2353456(2298K) committ= ed =3D 2490368(2432K) max =3D 1073741824(1048576K) INFO [main] 2018-01-10 10:43:30,337 CassandraDaemon.java:484 - Par Eden Sp= ace Heap memory: init =3D 322174976(314624K) used =3D 109586656(107018K) co= mmitted =3D 322174976(314624K) max =3D 322174976(314624K) INFO [main] 2018-01-10 10:43:30,338 CassandraDaemon.java:484 - Par Survivo= r Space Heap memory: init =3D 40239104(39296K) used =3D 0(0K) committed =3D= 40239104(39296K) max =3D 40239104(39296K) INFO [main] 2018-01-10 10:43:30,338 CassandraDaemon.java:484 - CMS Old Gen= Heap memory: init =3D 1207959552(1179648K) used =3D 0(0K) committed =3D 12= 07959552(1179648K) max =3D 1207959552(1179648K) INFO [main] 2018-01-10 10:43:30,338 CassandraDaemon.java:486 - Classpath: = /etc/cassandra:/usr/share/cassandra/lib/HdrHistogram-2.1.9.jar:/usr/share/c= assandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/sh= are/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-5.0.= 4.jar:/usr/share/cassandra/lib/caffeine-2.2.6.jar:/usr/share/cassandra/lib/= cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/commons-cli= -1.1.jar:/usr/share/cassandra/lib/commons-codec-1.9.jar:/usr/share/cassandr= a/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:= /usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/co= ncurrent-trees-2.4.0.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-l= ru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandr= a/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-18.0.jar:/usr/share/cass= andra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/hppc-0.5.4.jar:= /usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/li= b/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr= /share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m= .jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra= /lib/jctools-core-1.2.1.jar:/usr/share/cassandra/lib/jflex-1.6.0.jar:/usr/s= hare/cassandra/lib/jna-4.2.2.jar:/usr/share/cassandra/lib/joda-time-2.4.jar= :/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/jsta= ckjunit-0.0.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/c= assandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-cl= assic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/= cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar= :/usr/share/cassandra/lib/metrics-jvm-3.1.0.jar:/usr/share/cassandra/lib/me= trics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.44.Final.jar= :/usr/share/cassandra/lib/ohc-core-0.4.4.jar:/usr/share/cassandra/lib/ohc-c= ore-j8-0.4.4.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.3.jar:/u= sr/share/cassandra/lib/reporter-config3-3.0.3.jar:/usr/share/cassandra/lib/= sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cas= sandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.= jar:/usr/share/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/usr/share/ca= ssandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.j= ar:/usr/share/cassandra/apache-cassandra-3.11.1.jar:/usr/share/cassandra/ap= ache-cassandra-thrift-3.11.1.jar:/usr/share/cassandra/apache-cassandra.jar:= /usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar INFO [main] 2018-01-10 10:43:30,339 CassandraDaemon.java:488 - JVM Argumen= ts: [-Xloggc:/var/log/cassandra/gc.log, -ea, -XX:+UseThreadPriorities, -XX:= ThreadPriorityPolicy=3D42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:S= tringTableSize=3D1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+= UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.n= et.preferIPv4Stack=3Dtrue, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+= CMSParallelRemarkEnabled, -XX:SurvivorRatio=3D8, -XX:MaxTenuringThreshold= =3D1, -XX:CMSInitiatingOccupancyFraction=3D75, -XX:+UseCMSInitiatingOccupan= cyOnly, -XX:CMSWaitDuration=3D10000, -XX:+CMSParallelInitialMarkEnabled, -X= X:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDe= tails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistri= bution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX= :+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=3D10, -XX:GCLogFileSize=3D10= M, -Xms1536M, -Xmx1536M, -Xmn384M, -XX:+UseCondCardMark, -XX:CompileCommand= File=3D/etc/cassandra/hotspot_compiler, -javaagent:/usr/share/cassandra/lib= /jamm-0.3.0.jar, -Dcassandra.jmx.local.port=3D7199, -Dcom.sun.management.jm= xremote.authenticate=3Dfalse, -Dcom.sun.management.jmxremote.password.file= =3D/etc/cassandra/jmxremote.password, -Djava.library.path=3D/usr/share/cass= andra/lib/sigar-bin, -Dlogback.configurationFile=3Dlogback.xml, -Dcassandra= .logdir=3D/var/log/cassandra, -Dcassandra.storagedir=3D/var/lib/cassandra, = -Dcassandra-pidfile=3D/var/run/cassandra/cassandra.pid, -XX:HeapDumpPath=3D= /var/lib/cassandra/java_1515602606.hprof, -XX:ErrorFile=3D/var/lib/cassandr= a/hs_err_1515602606.log] INFO [main] 2018-01-10 10:43:30,598 NativeLibrary.java:174 - JNA mlockall = successful WARN [main] 2018-01-10 10:43:30,599 StartupChecks.java:133 - jemalloc shar= ed library could not be preloaded to speed up memory allocations WARN [main] 2018-01-10 10:43:30,599 StartupChecks.java:166 - JMX is not en= abled to receive remote connections. Please see cassandra-env.sh for more i= nfo. INFO [main] 2018-01-10 10:43:30,601 SigarLibrary.java:44 - Initializing SI= GAR library INFO [main] 2018-01-10 10:43:30,614 SigarLibrary.java:180 - Checked OS set= tings and found them configured for optimal performance. WARN [main] 2018-01-10 10:43:30,619 StartupChecks.java:271 - Maximum numbe= r of memory map areas per process (vm.max_map_count) 65530 is too low, reco= mmended value: 1048575, you can change it with sysctl. INFO [main] 2018-01-10 10:43:30,975 QueryProcessor.java:116 - Initialized = prepared statement caches with 10 MB (native) and 10 MB (Thrift) INFO [main] 2018-01-10 10:43:31,541 ColumnFamilyStore.java:408 - Initializ= ing system.IndexInfo INFO [SSTableBatchOpen:1] 2018-01-10 10:43:32,405 BufferPool.java:230 - Gl= obal buffer pool is enabled, when pool is exhausted (max is 374.000MiB) it = will allocate on heap INFO [main] 2018-01-10 10:43:32,653 CacheService.java:112 - Initializing k= ey cache with capacity of 74 MBs. INFO [main] 2018-01-10 10:43:32,662 CacheService.java:134 - Initializing r= ow cache with capacity of 0 MBs INFO [main] 2018-01-10 10:43:32,664 CacheService.java:163 - Initializing c= ounter cache with capacity of 37 MBs INFO [main] 2018-01-10 10:43:32,666 CacheService.java:174 - Scheduling cou= nter cache save to every 7200 seconds (going to save all keys). INFO [main] 2018-01-10 10:43:32,720 ColumnFamilyStore.java:408 - Initializ= ing system.batches INFO [main] 2018-01-10 10:43:32,747 ColumnFamilyStore.java:408 - Initializ= ing system.paxos INFO [main] 2018-01-10 10:43:32,763 ColumnFamilyStore.java:408 - Initializ= ing system.local INFO [main] 2018-01-10 10:43:32,791 ColumnFamilyStore.java:408 - Initializ= ing system.peers INFO [main] 2018-01-10 10:43:32,814 ColumnFamilyStore.java:408 - Initializ= ing system.peer_events INFO [main] 2018-01-10 10:43:32,821 ColumnFamilyStore.java:408 - Initializ= ing system.range_xfers INFO [main] 2018-01-10 10:43:32,830 ColumnFamilyStore.java:408 - Initializ= ing system.compaction_history INFO [main] 2018-01-10 10:43:32,852 ColumnFamilyStore.java:408 - Initializ= ing system.sstable_activity INFO [main] 2018-01-10 10:43:32,977 ColumnFamilyStore.java:408 - Initializ= ing system.size_estimates INFO [main] 2018-01-10 10:43:32,992 ColumnFamilyStore.java:408 - Initializ= ing system.available_ranges INFO [main] 2018-01-10 10:43:33,002 ColumnFamilyStore.java:408 - Initializ= ing system.transferred_ranges INFO [main] 2018-01-10 10:43:33,008 ColumnFamilyStore.java:408 - Initializ= ing system.views_builds_in_progress INFO [main] 2018-01-10 10:43:33,014 ColumnFamilyStore.java:408 - Initializ= ing system.built_views INFO [main] 2018-01-10 10:43:33,021 ColumnFamilyStore.java:408 - Initializ= ing system.hints INFO [main] 2018-01-10 10:43:33,028 ColumnFamilyStore.java:408 - Initializ= ing system.batchlog INFO [main] 2018-01-10 10:43:33,036 ColumnFamilyStore.java:408 - Initializ= ing system.prepared_statements INFO [main] 2018-01-10 10:43:33,049 ColumnFamilyStore.java:408 - Initializ= ing system.schema_keyspaces INFO [main] 2018-01-10 10:43:33,055 ColumnFamilyStore.java:408 - Initializ= ing system.schema_columnfamilies INFO [main] 2018-01-10 10:43:33,062 ColumnFamilyStore.java:408 - Initializ= ing system.schema_columns INFO [main] 2018-01-10 10:43:33,068 ColumnFamilyStore.java:408 - Initializ= ing system.schema_triggers INFO [main] 2018-01-10 10:43:33,074 ColumnFamilyStore.java:408 - Initializ= ing system.schema_usertypes INFO [main] 2018-01-10 10:43:33,081 ColumnFamilyStore.java:408 - Initializ= ing system.schema_functions INFO [main] 2018-01-10 10:43:33,088 ColumnFamilyStore.java:408 - Initializ= ing system.schema_aggregates INFO [main] 2018-01-10 10:43:33,090 ViewManager.java:137 - Not submitting = build tasks for views in keyspace system as storage service is not initiali= zed INFO [main] 2018-01-10 10:43:33,250 ApproximateTime.java:44 - Scheduling a= pproximate time-check task with a precision of 10 milliseconds INFO [main] 2018-01-10 10:43:33,307 ColumnFamilyStore.java:408 - Initializ= ing system_schema.keyspaces INFO [main] 2018-01-10 10:43:33,322 ColumnFamilyStore.java:408 - Initializ= ing system_schema.tables INFO [main] 2018-01-10 10:43:33,336 ColumnFamilyStore.java:408 - Initializ= ing system_schema.columns INFO [main] 2018-01-10 10:43:33,347 ColumnFamilyStore.java:408 - Initializ= ing system_schema.triggers INFO [main] 2018-01-10 10:43:33,353 ColumnFamilyStore.java:408 - Initializ= ing system_schema.dropped_columns INFO [main] 2018-01-10 10:43:33,359 ColumnFamilyStore.java:408 - Initializ= ing system_schema.views INFO [main] 2018-01-10 10:43:33,364 ColumnFamilyStore.java:408 - Initializ= ing system_schema.types INFO [main] 2018-01-10 10:43:33,370 ColumnFamilyStore.java:408 - Initializ= ing system_schema.functions INFO [main] 2018-01-10 10:43:33,376 ColumnFamilyStore.java:408 - Initializ= ing system_schema.aggregates INFO [main] 2018-01-10 10:43:33,382 ColumnFamilyStore.java:408 - Initializ= ing system_schema.indexes INFO [main] 2018-01-10 10:43:33,388 ViewManager.java:137 - Not submitting = build tasks for views in keyspace system_schema as storage service is not i= nitialized ERROR [MemtableFlushWriter:1] 2018-01-10 10:43:33,507 LogTransaction.java:2= 73 - Transaction log [mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log= in /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377] = indicates txn was not completed, trying to abort it now ERROR [MemtableFlushWriter:1] 2018-01-10 10:43:33,512 LogTransaction.java:2= 76 - Failed to abort transaction log [mc_txn_flush_6579bf70-f625-11e7-8e56-= 71d63c551064.log in /var/lib/cassandra/data/system/local-7ad54392bcdd35a684= 174e047860b377] java.lang.RuntimeException: java.nio.file.AccessDeniedException: /var/lib/c= assandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_65= 79bf70-f625-11e7-8e56-71d63c551064.log =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590) ~[ap= ache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.java:5= 71) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.java:8= 5) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogRepl= icaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaSet.= java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:324)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogFile.abort(LogFile.java:257) ~[a= pache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:98) ~[a= pache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogTransaction$TransactionTidier.ru= n(LogTransaction.java:274) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogTransaction$TransactionTidier.ti= dy(LogTransaction.java:258) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.jav= a:321) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.ja= va:199) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:119= ) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogTransaction.complete(LogTransact= ion.java:393) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogTransaction.doAbort(LogTransacti= on.java:410) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransacti= onal.abort(Transactional.java:144) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.doAbort(Lifecy= cleTransaction.java:243) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransacti= onal.abort(Transactional.java:144) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnF= amilyStore.java:1156) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore= .java:1102) [apache-cassandra-3.11.1.jar:3.11.1] =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1142) [na:1.8.0_131] =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:617) [na:1.8.0_131] =09at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocal= Deallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11= .1] =09at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/sys= tem/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e7-= 8e56-71d63c551064.log =09at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84= ) ~[na:1.8.0_131] =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)= ~[na:1.8.0_131] =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)= ~[na:1.8.0_131] =09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvid= er.java:214) ~[na:1.8.0_131] =09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvid= er.java:434) ~[na:1.8.0_131] =09at java.nio.file.Files.newOutputStream(Files.java:216) ~[na:1.8.0_131] =09at java.nio.file.Files.write(Files.java:3351) ~[na:1.8.0_131] =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) ~[ap= ache-cassandra-3.11.1.jar:3.11.1] =09... 27 common frames omitted ERROR [MemtableFlushWriter:1] 2018-01-10 10:43:33,515 LogTransaction.java:2= 23 - Unable to delete /var/lib/cassandra/data/system/local-7ad54392bcdd35a6= 84174e047860b377/mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log as i= t does not exist, see debug log file for stack trace ERROR [MemtablePostFlush:1] 2018-01-10 10:43:33,520 CassandraDaemon.java:22= 8 - Exception in thread Thread[MemtablePostFlush:1,5,main] java.lang.RuntimeException: java.nio.file.AccessDeniedException: /var/lib/c= assandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_65= 79bf70-f625-11e7-8e56-71d63c551064.log =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590) ~[ap= ache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.java:5= 71) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.java:8= 5) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogRepl= icaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaSet.= java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:324)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285) ~[apa= che-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransact= ion.java:136) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(Lifec= ycleTransaction.java:529) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.format.big.BigTableWriter.(BigT= ableWriter.java:81) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.op= en(BigFormat.java:92) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWr= iter.java:102) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.create(Simpl= eSSTableMultiWriter.java:119) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.createS= STableMultiWriter(AbstractCompactionStrategy.java:587) ~[apache-cassandra-3= .11.1.jar:3.11.1] =09at org.apache.cassandra.db.compaction.CompactionStrategyManager.createSS= TableMultiWriter(CompactionStrategyManager.java:920) ~[apache-cassandra-3.1= 1.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore.createSSTableMultiWriter(Co= lumnFamilyStore.java:515) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memt= able.java:509) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable$FlushRunnable.(Memtable.java:4= 48) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable$FlushRunnable.(Memtable.java:4= 25) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:303) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnF= amilyStore.java:1137) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore= .java:1102) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1142) ~[na:1.8.0_131] =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:617) [na:1.8.0_131] =09at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocal= Deallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11= .1] =09at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] =09Suppressed: java.lang.RuntimeException: java.nio.file.AccessDeniedExcept= ion: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/= mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log =09=09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590) ~= [apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.jav= a:571) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.jav= a:85) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogR= eplicaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaS= et.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:3= 24) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogFile.abort(LogFile.java:257) = ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:98) = ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogTransaction.doAbort(LogTransa= ction.java:410) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransa= ctional.abort(Transactional.java:144) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.doAbort(Lif= ecycleTransaction.java:243) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransa= ctional.abort(Transactional.java:144) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(Colu= mnFamilyStore.java:1156) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09... 5 common frames omitted =09Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/= system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11= e7-8e56-71d63c551064.log =09=09at sun.nio.fs.UnixException.translateToIOException(UnixException.java= :84) =09=09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:1= 02) =09=09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:1= 07) =09=09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemPro= vider.java:214) =09=09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemPro= vider.java:434) =09=09at java.nio.file.Files.newOutputStream(Files.java:216) =09=09at java.nio.file.Files.write(Files.java:3351) =09=09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) =09=09... 21 common frames omitted Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/sys= tem/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e7-= 8e56-71d63c551064.log =09at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84= ) ~[na:1.8.0_131] =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)= ~[na:1.8.0_131] =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)= ~[na:1.8.0_131] =09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvid= er.java:214) ~[na:1.8.0_131] =09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvid= er.java:434) ~[na:1.8.0_131] =09at java.nio.file.Files.newOutputStream(Files.java:216) ~[na:1.8.0_131] =09at java.nio.file.Files.write(Files.java:3351) ~[na:1.8.0_131] =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) ~[ap= ache-cassandra-3.11.1.jar:3.11.1] =09... 27 common frames omitted ERROR [main] 2018-01-10 10:43:33,522 CassandraDaemon.java:706 - Exception e= ncountered during startup java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.l= ang.RuntimeException: java.nio.file.AccessDeniedException: /var/lib/cassand= ra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70= -f625-11e7-8e56-71d63c551064.log =09at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:= 404) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.SystemKeyspace.forceBlockingFlush(SystemKeysp= ace.java:819) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.SystemKeyspace.removeTruncationRecord(SystemK= eyspace.java:670) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore.invalidate(ColumnFamilyStor= e.java:550) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore.invalidate(ColumnFamilyStor= e.java:526) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.schema.LegacySchemaMigrator.lambda$unloadLegacyS= chemaTables$1(LegacySchemaMigrator.java:137) ~[apache-cassandra-3.11.1.jar:= 3.11.1] =09at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_131] =09at org.apache.cassandra.schema.LegacySchemaMigrator.unloadLegacySchemaTa= bles(LegacySchemaMigrator.java:137) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.schema.LegacySchemaMigrator.migrate(LegacySchema= Migrator.java:83) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.ja= va:256) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon= .java:600) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.jav= a:689) [apache-cassandra-3.11.1.jar:3.11.1] Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeExcept= ion: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/system/lo= cal-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e7-8e56-7= 1d63c551064.log =09at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.= 0_131] =09at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_1= 31] =09at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:= 400) ~[apache-cassandra-3.11.1.jar:3.11.1] =09... 11 common frames omitted Caused by: java.lang.RuntimeException: java.nio.file.AccessDeniedException:= /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_t= xn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590) ~[ap= ache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.java:5= 71) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.java:8= 5) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogRepl= icaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaSet.= java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:324)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285) ~[apa= che-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransact= ion.java:136) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(Lifec= ycleTransaction.java:529) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.format.big.BigTableWriter.(BigT= ableWriter.java:81) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.op= en(BigFormat.java:92) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWr= iter.java:102) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.create(Simpl= eSSTableMultiWriter.java:119) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.createS= STableMultiWriter(AbstractCompactionStrategy.java:587) ~[apache-cassandra-3= .11.1.jar:3.11.1] =09at org.apache.cassandra.db.compaction.CompactionStrategyManager.createSS= TableMultiWriter(CompactionStrategyManager.java:920) ~[apache-cassandra-3.1= 1.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore.createSSTableMultiWriter(Co= lumnFamilyStore.java:515) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memt= able.java:509) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable$FlushRunnable.(Memtable.java:4= 48) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable$FlushRunnable.(Memtable.java:4= 25) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:303) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnF= amilyStore.java:1137) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore= .java:1102) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1142) ~[na:1.8.0_131] =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:617) ~[na:1.8.0_131] =09at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocal= Deallocator$0(NamedThreadFactory.java:81) ~[apache-cassandra-3.11.1.jar:3.1= 1.1] =09at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] =09Suppressed: java.lang.RuntimeException: java.nio.file.AccessDeniedExcept= ion: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/= mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log =09=09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590) ~= [apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.jav= a:571) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.jav= a:85) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogR= eplicaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaS= et.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:3= 24) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogFile.abort(LogFile.java:257) = ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:98) = ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogTransaction.doAbort(LogTransa= ction.java:410) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransa= ctional.abort(Transactional.java:144) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.doAbort(Lif= ecycleTransaction.java:243) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransa= ctional.abort(Transactional.java:144) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(Colu= mnFamilyStore.java:1156) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09... 5 common frames omitted =09Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/= system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11= e7-8e56-71d63c551064.log =09=09at sun.nio.fs.UnixException.translateToIOException(UnixException.java= :84) =09=09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:1= 02) =09=09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:1= 07) =09=09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemPro= vider.java:214) =09=09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemPro= vider.java:434) =09=09at java.nio.file.Files.newOutputStream(Files.java:216) =09=09at java.nio.file.Files.write(Files.java:3351) =09=09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) =09=09... 21 common frames omitted Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/sys= tem/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e7-= 8e56-71d63c551064.log =09at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84= ) ~[na:1.8.0_131] =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)= ~[na:1.8.0_131] =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)= ~[na:1.8.0_131] =09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvid= er.java:214) ~[na:1.8.0_131] =09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvid= er.java:434) ~[na:1.8.0_131] =09at java.nio.file.Files.newOutputStream(Files.java:216) ~[na:1.8.0_131] =09at java.nio.file.Files.write(Files.java:3351) ~[na:1.8.0_131] =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) ~[ap= ache-cassandra-3.11.1.jar:3.11.1] =09... 27 common frames omitted {code} was: this exception cuts the node starting: {code} java.lang.RuntimeException: java.nio.file.AccessDeniedException: /var/lib/c= assandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_65= 79bf70-f625-11e7-8e56-71d63c551064.log the offending file doesn't exists: ls -la /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b3= 77/ total 160 drwxr-xr-x 3 cassandra cassandra 4096 ene 7 11:48 . drwxr-xr-x 26 cassandra cassandra 4096 dic 18 07:30 .. drwxr-xr-x 2 cassandra cassandra 4096 dic 18 07:30 backups -rw-r--r-- 1 cassandra cassandra 43 ene 7 03:50 mc-100-big-CompressionI= nfo.db -rw-r--r-- 1 cassandra cassandra 52 ene 7 03:50 mc-100-big-Data.db -rw-r--r-- 1 cassandra cassandra 8 ene 7 03:50 mc-100-big-Digest.crc32 -rw-r--r-- 1 cassandra cassandra 16 ene 7 03:50 mc-100-big-Filter.db -rw-r--r-- 1 cassandra cassandra 9 ene 7 03:50 mc-100-big-Index.db -rw-r--r-- 1 cassandra cassandra 4615 ene 7 03:50 mc-100-big-Statistics.d= b -rw-r--r-- 1 cassandra cassandra 59 ene 7 03:50 mc-100-big-Summary.db -rw-r--r-- 1 cassandra cassandra 92 ene 7 03:50 mc-100-big-TOC.txt -rw-r--r-- 1 cassandra cassandra 43 ene 7 11:48 mc-101-big-CompressionI= nfo.db -rw-r--r-- 1 cassandra cassandra 50 ene 7 11:48 mc-101-big-Data.db -rw-r--r-- 1 cassandra cassandra 10 ene 7 11:48 mc-101-big-Digest.crc32 -rw-r--r-- 1 cassandra cassandra 16 ene 7 11:48 mc-101-big-Filter.db -rw-r--r-- 1 cassandra cassandra 9 ene 7 11:48 mc-101-big-Index.db -rw-r--r-- 1 cassandra cassandra 4615 ene 7 11:48 mc-101-big-Statistics.d= b -rw-r--r-- 1 cassandra cassandra 59 ene 7 11:48 mc-101-big-Summary.db -rw-r--r-- 1 cassandra cassandra 92 ene 7 11:48 mc-101-big-TOC.txt -rw-r--r-- 1 cassandra cassandra 51 ene 6 17:50 mc-98-big-CompressionIn= fo.db -rw-r--r-- 1 cassandra cassandra 5226 ene 6 17:50 mc-98-big-Data.db -rw-r--r-- 1 cassandra cassandra 10 ene 6 17:50 mc-98-big-Digest.crc32 -rw-r--r-- 1 cassandra cassandra 16 ene 6 17:50 mc-98-big-Filter.db -rw-r--r-- 1 cassandra cassandra 9 ene 6 17:50 mc-98-big-Index.db -rw-r--r-- 1 cassandra cassandra 5654 ene 6 17:50 mc-98-big-Statistics.db -rw-r--r-- 1 cassandra cassandra 59 ene 6 17:50 mc-98-big-Summary.db -rw-r--r-- 1 cassandra cassandra 92 ene 6 17:50 mc-98-big-TOC.txt -rw-r--r-- 1 cassandra cassandra 43 ene 7 02:50 mc-99-big-CompressionIn= fo.db -rw-r--r-- 1 cassandra cassandra 50 ene 7 02:50 mc-99-big-Data.db -rw-r--r-- 1 cassandra cassandra 10 ene 7 02:50 mc-99-big-Digest.crc32 -rw-r--r-- 1 cassandra cassandra 16 ene 7 02:50 mc-99-big-Filter.db -rw-r--r-- 1 cassandra cassandra 9 ene 7 02:50 mc-99-big-Index.db -rw-r--r-- 1 cassandra cassandra 4615 ene 7 02:50 mc-99-big-Statistics.db -rw-r--r-- 1 cassandra cassandra 59 ene 7 02:50 mc-99-big-Summary.db -rw-r--r-- 1 cassandra cassandra 92 ene 7 02:50 mc-99-big-TOC.txt {code} permissions are ok, everything assigned to cassandra:cassandra the system.log: {code} INFO [main] 2018-01-10 10:43:29,367 YamlConfigurationLoader.java:89 - Conf= iguration location: file:/etc/cassandra/cassandra.yaml INFO [main] 2018-01-10 10:43:29,887 Config.java:481 - Node configuration:[= allocate_tokens_for_keyspace=3Dnull; authenticator=3DAllowAllAuthenticator;= authorizer=3DAllowAllAuthorizer; auto_bootstrap=3Dtrue; auto_snapshot=3Dtr= ue; back_pressure_enabled=3Dfalse; back_pressure_strategy=3Dnull; batch_siz= e_fail_threshold_in_kb=3D500; batch_size_warn_threshold_in_kb=3D50; batchlo= g_replay_throttle_in_kb=3D1024; broadcast_address=3Dnull; broadcast_rpc_add= ress=3Dnull; buffer_pool_use_heap_if_exhausted=3Dtrue; cas_contention_timeo= ut_in_ms=3D10000; cdc_enabled=3Dfalse; cdc_free_space_check_interval_ms=3D2= 50; cdc_raw_directory=3Dnull; cdc_total_space_in_mb=3D0; client_encryption_= options=3D; cluster_name=3DPAP Cluster; column_index_cache_size_i= n_kb=3D2; column_index_size_in_kb=3D64; commit_failure_policy=3Dstop; commi= tlog_compression=3Dnull; commitlog_directory=3D/var/lib/cassandra/commitlog= ; commitlog_max_compression_buffers_in_pool=3D3; commitlog_periodic_queue_s= ize=3D-1; commitlog_segment_size_in_mb=3D32; commitlog_sync=3Dperiodic; com= mitlog_sync_batch_window_in_ms=3DNaN; commitlog_sync_period_in_ms=3D10000; = commitlog_total_space_in_mb=3Dnull; compaction_large_partition_warning_thre= shold_mb=3D100; compaction_throughput_mb_per_sec=3D16; concurrent_compactor= s=3Dnull; concurrent_counter_writes=3D32; concurrent_materialized_view_writ= es=3D32; concurrent_reads=3D32; concurrent_replicates=3Dnull; concurrent_wr= ites=3D32; counter_cache_keys_to_save=3D2147483647; counter_cache_save_peri= od=3D7200; counter_cache_size_in_mb=3Dnull; counter_write_request_timeout_i= n_ms=3D50000; credentials_cache_max_entries=3D1000; credentials_update_inte= rval_in_ms=3D-1; credentials_validity_in_ms=3D2000; cross_node_timeout=3Dfa= lse; data_file_directories=3D[Ljava.lang.String;@79ca92b9; disk_access_mode= =3Dauto; disk_failure_policy=3Dstop; disk_optimization_estimate_percentile= =3D0.95; disk_optimization_page_cross_chance=3D0.1; disk_optimization_strat= egy=3Dssd; dynamic_snitch=3Dtrue; dynamic_snitch_badness_threshold=3D0.1; d= ynamic_snitch_reset_interval_in_ms=3D600000; dynamic_snitch_update_interval= _in_ms=3D100; enable_scripted_user_defined_functions=3Dfalse; enable_user_d= efined_functions=3Dfalse; enable_user_defined_functions_threads=3Dtrue; enc= ryption_options=3Dnull; endpoint_snitch=3DGossipingPropertyFileSnitch; file= _cache_size_in_mb=3Dnull; gc_log_threshold_in_ms=3D200; gc_warn_threshold_i= n_ms=3D1000; hinted_handoff_disabled_datacenters=3D[]; hinted_handoff_enabl= ed=3Dtrue; hinted_handoff_throttle_in_kb=3D1024; hints_compression=3Dnull; = hints_directory=3Dnull; hints_flush_period_in_ms=3D10000; incremental_backu= ps=3Dfalse; index_interval=3Dnull; index_summary_capacity_in_mb=3Dnull; ind= ex_summary_resize_interval_in_minutes=3D60; initial_token=3Dnull; inter_dc_= stream_throughput_outbound_megabits_per_sec=3D200; inter_dc_tcp_nodelay=3Df= alse; internode_authenticator=3Dnull; internode_compression=3Dall; internod= e_recv_buff_size_in_bytes=3D0; internode_send_buff_size_in_bytes=3D0; key_c= ache_keys_to_save=3D2147483647; key_cache_save_period=3D14400; key_cache_si= ze_in_mb=3Dnull; listen_address=3D192.121.166.40; listen_interface=3Dnull; = listen_interface_prefer_ipv6=3Dfalse; listen_on_broadcast_address=3Dfalse; = max_hint_window_in_ms=3D10800000; max_hints_delivery_threads=3D2; max_hints= _file_size_in_mb=3D128; max_mutation_size_in_kb=3Dnull; max_streaming_retri= es=3D3; max_value_size_in_mb=3D256; memtable_allocation_type=3Dheap_buffers= ; memtable_cleanup_threshold=3Dnull; memtable_flush_writers=3D0; memtable_h= eap_space_in_mb=3Dnull; memtable_offheap_space_in_mb=3Dnull; min_free_space= _per_drive_in_mb=3D50; native_transport_max_concurrent_connections=3D-1; na= tive_transport_max_concurrent_connections_per_ip=3D-1; native_transport_max= _frame_size_in_mb=3D256; native_transport_max_threads=3D128; native_transpo= rt_port=3D9042; native_transport_port_ssl=3Dnull; num_tokens=3D256; otc_bac= klog_expiration_interval_ms=3D200; otc_coalescing_enough_coalesced_messages= =3D8; otc_coalescing_strategy=3DDISABLED; otc_coalescing_window_us=3D200; p= artitioner=3Dorg.apache.cassandra.dht.Murmur3Partitioner; permissions_cache= _max_entries=3D1000; permissions_update_interval_in_ms=3D-1; permissions_va= lidity_in_ms=3D2000; phi_convict_threshold=3D8.0; prepared_statements_cache= _size_mb=3Dnull; range_request_timeout_in_ms=3D100000; read_request_timeout= _in_ms=3D100000; request_scheduler=3Dorg.apache.cassandra.scheduler.NoSched= uler; request_scheduler_id=3Dnull; request_scheduler_options=3Dnull; reques= t_timeout_in_ms=3D100000; role_manager=3DCassandraRoleManager; roles_cache_= max_entries=3D1000; roles_update_interval_in_ms=3D-1; roles_validity_in_ms= =3D2000; row_cache_class_name=3Dorg.apache.cassandra.cache.OHCProvider; row= _cache_keys_to_save=3D2147483647; row_cache_save_period=3D0; row_cache_size= _in_mb=3D0; rpc_address=3Dlocalhost; rpc_interface=3Dnull; rpc_interface_pr= efer_ipv6=3Dfalse; rpc_keepalive=3Dtrue; rpc_listen_backlog=3D50; rpc_max_t= hreads=3D2147483647; rpc_min_threads=3D16; rpc_port=3D9160; rpc_recv_buff_s= ize_in_bytes=3Dnull; rpc_send_buff_size_in_bytes=3Dnull; rpc_server_type=3D= sync; saved_caches_directory=3D/var/lib/cassandra/saved_caches; seed_provid= er=3Dorg.apache.cassandra.locator.SimpleSeedProvider{seeds=3D63.142.255.107= ,37.247.116.68}; server_encryption_options=3D; slow_query_log_tim= eout_in_ms=3D500; snapshot_before_compaction=3Dfalse; ssl_storage_port=3D70= 71; sstable_preemptive_open_interval_in_mb=3D50; start_native_transport=3Dt= rue; start_rpc=3Dfalse; storage_port=3D7070; stream_throughput_outbound_meg= abits_per_sec=3D200; streaming_keep_alive_period_in_secs=3D300; streaming_s= ocket_timeout_in_ms=3D86400000; thrift_framed_transport_size_in_mb=3D15; th= rift_max_message_length_in_mb=3D16; thrift_prepared_statements_cache_size_m= b=3Dnull; tombstone_failure_threshold=3D100000; tombstone_warn_threshold=3D= 1000; tracetype_query_ttl=3D86400; tracetype_repair_ttl=3D604800; transpare= nt_data_encryption_options=3Dorg.apache.cassandra.config.TransparentDataEnc= ryptionOptions@1460a8c0; trickle_fsync=3Dfalse; trickle_fsync_interval_in_k= b=3D10240; truncate_request_timeout_in_ms=3D600000; unlogged_batch_across_p= artitions_warn_threshold=3D10; user_defined_function_fail_timeout=3D1500; u= ser_defined_function_warn_timeout=3D500; user_function_timeout_policy=3Ddie= ; windows_timer_interval=3D1; write_request_timeout_in_ms=3D50000] INFO [main] 2018-01-10 10:43:29,888 DatabaseDescriptor.java:367 - DiskAcce= ssMode 'auto' determined to be mmap, indexAccessMode is mmap INFO [main] 2018-01-10 10:43:29,888 DatabaseDescriptor.java:421 - Global m= emtable on-heap threshold is enabled at 374MB INFO [main] 2018-01-10 10:43:29,888 DatabaseDescriptor.java:425 - Global m= emtable off-heap threshold is enabled at 374MB WARN [main] 2018-01-10 10:43:29,897 DatabaseDescriptor.java:470 - Small co= mmitlog volume detected at /var/lib/cassandra/commitlog; setting commitlog_= total_space_in_mb to 7680. You can override this in cassandra.yaml WARN [main] 2018-01-10 10:43:29,899 DatabaseDescriptor.java:497 - Small cd= c volume detected at /var/lib/cassandra/cdc_raw; setting cdc_total_space_in= _mb to 3840. You can override this in cassandra.yaml WARN [main] 2018-01-10 10:43:30,022 DatabaseDescriptor.java:546 - Only 18.= 837GiB free across all data volumes. Consider adding more capacity to your = cluster or removing obsolete snapshots INFO [main] 2018-01-10 10:43:30,066 RateBasedBackPressure.java:123 - Initi= alized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window si= ze: 50000. INFO [main] 2018-01-10 10:43:30,066 DatabaseDescriptor.java:725 - Back-pre= ssure is disabled with strategy null. INFO [main] 2018-01-10 10:43:30,250 GossipingPropertyFileSnitch.java:64 - = Loaded cassandra-topology.properties for compatibility INFO [main] 2018-01-10 10:43:30,329 JMXServerUtils.java:249 - Configured J= MX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi INFO [main] 2018-01-10 10:43:30,335 CassandraDaemon.java:471 - Hostname: u= k1.plugandplay.click INFO [main] 2018-01-10 10:43:30,335 CassandraDaemon.java:478 - JVM vendor/= version: Java HotSpot(TM) 64-Bit Server VM/1.8.0_131 INFO [main] 2018-01-10 10:43:30,336 CassandraDaemon.java:479 - Heap size: = 1.463GiB/1.463GiB INFO [main] 2018-01-10 10:43:30,336 CassandraDaemon.java:484 - Code Cache = Non-heap memory: init =3D 2555904(2496K) used =3D 4545344(4438K) committed = =3D 4587520(4480K) max =3D 251658240(245760K) INFO [main] 2018-01-10 10:43:30,337 CassandraDaemon.java:484 - Metaspace N= on-heap memory: init =3D 0(0K) used =3D 19068664(18621K) committed =3D 1979= 1872(19328K) max =3D -1(-1K) INFO [main] 2018-01-10 10:43:30,337 CassandraDaemon.java:484 - Compressed = Class Space Non-heap memory: init =3D 0(0K) used =3D 2353456(2298K) committ= ed =3D 2490368(2432K) max =3D 1073741824(1048576K) INFO [main] 2018-01-10 10:43:30,337 CassandraDaemon.java:484 - Par Eden Sp= ace Heap memory: init =3D 322174976(314624K) used =3D 109586656(107018K) co= mmitted =3D 322174976(314624K) max =3D 322174976(314624K) INFO [main] 2018-01-10 10:43:30,338 CassandraDaemon.java:484 - Par Survivo= r Space Heap memory: init =3D 40239104(39296K) used =3D 0(0K) committed =3D= 40239104(39296K) max =3D 40239104(39296K) INFO [main] 2018-01-10 10:43:30,338 CassandraDaemon.java:484 - CMS Old Gen= Heap memory: init =3D 1207959552(1179648K) used =3D 0(0K) committed =3D 12= 07959552(1179648K) max =3D 1207959552(1179648K) INFO [main] 2018-01-10 10:43:30,338 CassandraDaemon.java:486 - Classpath: = /etc/cassandra:/usr/share/cassandra/lib/HdrHistogram-2.1.9.jar:/usr/share/c= assandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/sh= are/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-5.0.= 4.jar:/usr/share/cassandra/lib/caffeine-2.2.6.jar:/usr/share/cassandra/lib/= cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/commons-cli= -1.1.jar:/usr/share/cassandra/lib/commons-codec-1.9.jar:/usr/share/cassandr= a/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:= /usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/co= ncurrent-trees-2.4.0.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-l= ru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandr= a/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-18.0.jar:/usr/share/cass= andra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/hppc-0.5.4.jar:= /usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/li= b/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr= /share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m= .jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra= /lib/jctools-core-1.2.1.jar:/usr/share/cassandra/lib/jflex-1.6.0.jar:/usr/s= hare/cassandra/lib/jna-4.2.2.jar:/usr/share/cassandra/lib/joda-time-2.4.jar= :/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/jsta= ckjunit-0.0.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/c= assandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-cl= assic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/= cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar= :/usr/share/cassandra/lib/metrics-jvm-3.1.0.jar:/usr/share/cassandra/lib/me= trics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.44.Final.jar= :/usr/share/cassandra/lib/ohc-core-0.4.4.jar:/usr/share/cassandra/lib/ohc-c= ore-j8-0.4.4.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.3.jar:/u= sr/share/cassandra/lib/reporter-config3-3.0.3.jar:/usr/share/cassandra/lib/= sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cas= sandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.= jar:/usr/share/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/usr/share/ca= ssandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.j= ar:/usr/share/cassandra/apache-cassandra-3.11.1.jar:/usr/share/cassandra/ap= ache-cassandra-thrift-3.11.1.jar:/usr/share/cassandra/apache-cassandra.jar:= /usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar INFO [main] 2018-01-10 10:43:30,339 CassandraDaemon.java:488 - JVM Argumen= ts: [-Xloggc:/var/log/cassandra/gc.log, -ea, -XX:+UseThreadPriorities, -XX:= ThreadPriorityPolicy=3D42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:S= tringTableSize=3D1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+= UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.n= et.preferIPv4Stack=3Dtrue, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+= CMSParallelRemarkEnabled, -XX:SurvivorRatio=3D8, -XX:MaxTenuringThreshold= =3D1, -XX:CMSInitiatingOccupancyFraction=3D75, -XX:+UseCMSInitiatingOccupan= cyOnly, -XX:CMSWaitDuration=3D10000, -XX:+CMSParallelInitialMarkEnabled, -X= X:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDe= tails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistri= bution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX= :+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=3D10, -XX:GCLogFileSize=3D10= M, -Xms1536M, -Xmx1536M, -Xmn384M, -XX:+UseCondCardMark, -XX:CompileCommand= File=3D/etc/cassandra/hotspot_compiler, -javaagent:/usr/share/cassandra/lib= /jamm-0.3.0.jar, -Dcassandra.jmx.local.port=3D7199, -Dcom.sun.management.jm= xremote.authenticate=3Dfalse, -Dcom.sun.management.jmxremote.password.file= =3D/etc/cassandra/jmxremote.password, -Djava.library.path=3D/usr/share/cass= andra/lib/sigar-bin, -Dlogback.configurationFile=3Dlogback.xml, -Dcassandra= .logdir=3D/var/log/cassandra, -Dcassandra.storagedir=3D/var/lib/cassandra, = -Dcassandra-pidfile=3D/var/run/cassandra/cassandra.pid, -XX:HeapDumpPath=3D= /var/lib/cassandra/java_1515602606.hprof, -XX:ErrorFile=3D/var/lib/cassandr= a/hs_err_1515602606.log] INFO [main] 2018-01-10 10:43:30,598 NativeLibrary.java:174 - JNA mlockall = successful WARN [main] 2018-01-10 10:43:30,599 StartupChecks.java:133 - jemalloc shar= ed library could not be preloaded to speed up memory allocations WARN [main] 2018-01-10 10:43:30,599 StartupChecks.java:166 - JMX is not en= abled to receive remote connections. Please see cassandra-env.sh for more i= nfo. INFO [main] 2018-01-10 10:43:30,601 SigarLibrary.java:44 - Initializing SI= GAR library INFO [main] 2018-01-10 10:43:30,614 SigarLibrary.java:180 - Checked OS set= tings and found them configured for optimal performance. WARN [main] 2018-01-10 10:43:30,619 StartupChecks.java:271 - Maximum numbe= r of memory map areas per process (vm.max_map_count) 65530 is too low, reco= mmended value: 1048575, you can change it with sysctl. INFO [main] 2018-01-10 10:43:30,975 QueryProcessor.java:116 - Initialized = prepared statement caches with 10 MB (native) and 10 MB (Thrift) INFO [main] 2018-01-10 10:43:31,541 ColumnFamilyStore.java:408 - Initializ= ing system.IndexInfo INFO [SSTableBatchOpen:1] 2018-01-10 10:43:32,405 BufferPool.java:230 - Gl= obal buffer pool is enabled, when pool is exhausted (max is 374.000MiB) it = will allocate on heap INFO [main] 2018-01-10 10:43:32,653 CacheService.java:112 - Initializing k= ey cache with capacity of 74 MBs. INFO [main] 2018-01-10 10:43:32,662 CacheService.java:134 - Initializing r= ow cache with capacity of 0 MBs INFO [main] 2018-01-10 10:43:32,664 CacheService.java:163 - Initializing c= ounter cache with capacity of 37 MBs INFO [main] 2018-01-10 10:43:32,666 CacheService.java:174 - Scheduling cou= nter cache save to every 7200 seconds (going to save all keys). INFO [main] 2018-01-10 10:43:32,720 ColumnFamilyStore.java:408 - Initializ= ing system.batches INFO [main] 2018-01-10 10:43:32,747 ColumnFamilyStore.java:408 - Initializ= ing system.paxos INFO [main] 2018-01-10 10:43:32,763 ColumnFamilyStore.java:408 - Initializ= ing system.local INFO [main] 2018-01-10 10:43:32,791 ColumnFamilyStore.java:408 - Initializ= ing system.peers INFO [main] 2018-01-10 10:43:32,814 ColumnFamilyStore.java:408 - Initializ= ing system.peer_events INFO [main] 2018-01-10 10:43:32,821 ColumnFamilyStore.java:408 - Initializ= ing system.range_xfers INFO [main] 2018-01-10 10:43:32,830 ColumnFamilyStore.java:408 - Initializ= ing system.compaction_history INFO [main] 2018-01-10 10:43:32,852 ColumnFamilyStore.java:408 - Initializ= ing system.sstable_activity INFO [main] 2018-01-10 10:43:32,977 ColumnFamilyStore.java:408 - Initializ= ing system.size_estimates INFO [main] 2018-01-10 10:43:32,992 ColumnFamilyStore.java:408 - Initializ= ing system.available_ranges INFO [main] 2018-01-10 10:43:33,002 ColumnFamilyStore.java:408 - Initializ= ing system.transferred_ranges INFO [main] 2018-01-10 10:43:33,008 ColumnFamilyStore.java:408 - Initializ= ing system.views_builds_in_progress INFO [main] 2018-01-10 10:43:33,014 ColumnFamilyStore.java:408 - Initializ= ing system.built_views INFO [main] 2018-01-10 10:43:33,021 ColumnFamilyStore.java:408 - Initializ= ing system.hints INFO [main] 2018-01-10 10:43:33,028 ColumnFamilyStore.java:408 - Initializ= ing system.batchlog INFO [main] 2018-01-10 10:43:33,036 ColumnFamilyStore.java:408 - Initializ= ing system.prepared_statements INFO [main] 2018-01-10 10:43:33,049 ColumnFamilyStore.java:408 - Initializ= ing system.schema_keyspaces INFO [main] 2018-01-10 10:43:33,055 ColumnFamilyStore.java:408 - Initializ= ing system.schema_columnfamilies INFO [main] 2018-01-10 10:43:33,062 ColumnFamilyStore.java:408 - Initializ= ing system.schema_columns INFO [main] 2018-01-10 10:43:33,068 ColumnFamilyStore.java:408 - Initializ= ing system.schema_triggers INFO [main] 2018-01-10 10:43:33,074 ColumnFamilyStore.java:408 - Initializ= ing system.schema_usertypes INFO [main] 2018-01-10 10:43:33,081 ColumnFamilyStore.java:408 - Initializ= ing system.schema_functions INFO [main] 2018-01-10 10:43:33,088 ColumnFamilyStore.java:408 - Initializ= ing system.schema_aggregates INFO [main] 2018-01-10 10:43:33,090 ViewManager.java:137 - Not submitting = build tasks for views in keyspace system as storage service is not initiali= zed INFO [main] 2018-01-10 10:43:33,250 ApproximateTime.java:44 - Scheduling a= pproximate time-check task with a precision of 10 milliseconds INFO [main] 2018-01-10 10:43:33,307 ColumnFamilyStore.java:408 - Initializ= ing system_schema.keyspaces INFO [main] 2018-01-10 10:43:33,322 ColumnFamilyStore.java:408 - Initializ= ing system_schema.tables INFO [main] 2018-01-10 10:43:33,336 ColumnFamilyStore.java:408 - Initializ= ing system_schema.columns INFO [main] 2018-01-10 10:43:33,347 ColumnFamilyStore.java:408 - Initializ= ing system_schema.triggers INFO [main] 2018-01-10 10:43:33,353 ColumnFamilyStore.java:408 - Initializ= ing system_schema.dropped_columns INFO [main] 2018-01-10 10:43:33,359 ColumnFamilyStore.java:408 - Initializ= ing system_schema.views INFO [main] 2018-01-10 10:43:33,364 ColumnFamilyStore.java:408 - Initializ= ing system_schema.types INFO [main] 2018-01-10 10:43:33,370 ColumnFamilyStore.java:408 - Initializ= ing system_schema.functions INFO [main] 2018-01-10 10:43:33,376 ColumnFamilyStore.java:408 - Initializ= ing system_schema.aggregates INFO [main] 2018-01-10 10:43:33,382 ColumnFamilyStore.java:408 - Initializ= ing system_schema.indexes INFO [main] 2018-01-10 10:43:33,388 ViewManager.java:137 - Not submitting = build tasks for views in keyspace system_schema as storage service is not i= nitialized ERROR [MemtableFlushWriter:1] 2018-01-10 10:43:33,507 LogTransaction.java:2= 73 - Transaction log [mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log= in /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377] = indicates txn was not completed, trying to abort it now ERROR [MemtableFlushWriter:1] 2018-01-10 10:43:33,512 LogTransaction.java:2= 76 - Failed to abort transaction log [mc_txn_flush_6579bf70-f625-11e7-8e56-= 71d63c551064.log in /var/lib/cassandra/data/system/local-7ad54392bcdd35a684= 174e047860b377] java.lang.RuntimeException: java.nio.file.AccessDeniedException: /var/lib/c= assandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_65= 79bf70-f625-11e7-8e56-71d63c551064.log =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590) ~[ap= ache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.java:5= 71) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.java:8= 5) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogRepl= icaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaSet.= java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:324)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogFile.abort(LogFile.java:257) ~[a= pache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:98) ~[a= pache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogTransaction$TransactionTidier.ru= n(LogTransaction.java:274) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogTransaction$TransactionTidier.ti= dy(LogTransaction.java:258) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.jav= a:321) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.ja= va:199) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:119= ) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogTransaction.complete(LogTransact= ion.java:393) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogTransaction.doAbort(LogTransacti= on.java:410) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransacti= onal.abort(Transactional.java:144) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.doAbort(Lifecy= cleTransaction.java:243) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransacti= onal.abort(Transactional.java:144) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnF= amilyStore.java:1156) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore= .java:1102) [apache-cassandra-3.11.1.jar:3.11.1] =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1142) [na:1.8.0_131] =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:617) [na:1.8.0_131] =09at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocal= Deallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11= .1] =09at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/sys= tem/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e7-= 8e56-71d63c551064.log =09at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84= ) ~[na:1.8.0_131] =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)= ~[na:1.8.0_131] =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)= ~[na:1.8.0_131] =09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvid= er.java:214) ~[na:1.8.0_131] =09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvid= er.java:434) ~[na:1.8.0_131] =09at java.nio.file.Files.newOutputStream(Files.java:216) ~[na:1.8.0_131] =09at java.nio.file.Files.write(Files.java:3351) ~[na:1.8.0_131] =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) ~[ap= ache-cassandra-3.11.1.jar:3.11.1] =09... 27 common frames omitted ERROR [MemtableFlushWriter:1] 2018-01-10 10:43:33,515 LogTransaction.java:2= 23 - Unable to delete /var/lib/cassandra/data/system/local-7ad54392bcdd35a6= 84174e047860b377/mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log as i= t does not exist, see debug log file for stack trace ERROR [MemtablePostFlush:1] 2018-01-10 10:43:33,520 CassandraDaemon.java:22= 8 - Exception in thread Thread[MemtablePostFlush:1,5,main] java.lang.RuntimeException: java.nio.file.AccessDeniedException: /var/lib/c= assandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_65= 79bf70-f625-11e7-8e56-71d63c551064.log =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590) ~[ap= ache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.java:5= 71) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.java:8= 5) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogRepl= icaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaSet.= java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:324)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285) ~[apa= che-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransact= ion.java:136) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(Lifec= ycleTransaction.java:529) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.format.big.BigTableWriter.(BigT= ableWriter.java:81) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.op= en(BigFormat.java:92) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWr= iter.java:102) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.create(Simpl= eSSTableMultiWriter.java:119) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.createS= STableMultiWriter(AbstractCompactionStrategy.java:587) ~[apache-cassandra-3= .11.1.jar:3.11.1] =09at org.apache.cassandra.db.compaction.CompactionStrategyManager.createSS= TableMultiWriter(CompactionStrategyManager.java:920) ~[apache-cassandra-3.1= 1.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore.createSSTableMultiWriter(Co= lumnFamilyStore.java:515) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memt= able.java:509) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable$FlushRunnable.(Memtable.java:4= 48) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable$FlushRunnable.(Memtable.java:4= 25) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:303) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnF= amilyStore.java:1137) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore= .java:1102) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1142) ~[na:1.8.0_131] =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:617) [na:1.8.0_131] =09at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocal= Deallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11= .1] =09at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] =09Suppressed: java.lang.RuntimeException: java.nio.file.AccessDeniedExcept= ion: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/= mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log =09=09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590) ~= [apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.jav= a:571) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.jav= a:85) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogR= eplicaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaS= et.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:3= 24) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogFile.abort(LogFile.java:257) = ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:98) = ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogTransaction.doAbort(LogTransa= ction.java:410) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransa= ctional.abort(Transactional.java:144) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.doAbort(Lif= ecycleTransaction.java:243) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransa= ctional.abort(Transactional.java:144) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(Colu= mnFamilyStore.java:1156) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09... 5 common frames omitted =09Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/= system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11= e7-8e56-71d63c551064.log =09=09at sun.nio.fs.UnixException.translateToIOException(UnixException.java= :84) =09=09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:1= 02) =09=09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:1= 07) =09=09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemPro= vider.java:214) =09=09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemPro= vider.java:434) =09=09at java.nio.file.Files.newOutputStream(Files.java:216) =09=09at java.nio.file.Files.write(Files.java:3351) =09=09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) =09=09... 21 common frames omitted Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/sys= tem/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e7-= 8e56-71d63c551064.log =09at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84= ) ~[na:1.8.0_131] =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)= ~[na:1.8.0_131] =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)= ~[na:1.8.0_131] =09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvid= er.java:214) ~[na:1.8.0_131] =09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvid= er.java:434) ~[na:1.8.0_131] =09at java.nio.file.Files.newOutputStream(Files.java:216) ~[na:1.8.0_131] =09at java.nio.file.Files.write(Files.java:3351) ~[na:1.8.0_131] =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) ~[ap= ache-cassandra-3.11.1.jar:3.11.1] =09... 27 common frames omitted ERROR [main] 2018-01-10 10:43:33,522 CassandraDaemon.java:706 - Exception e= ncountered during startup java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.l= ang.RuntimeException: java.nio.file.AccessDeniedException: /var/lib/cassand= ra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70= -f625-11e7-8e56-71d63c551064.log =09at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:= 404) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.SystemKeyspace.forceBlockingFlush(SystemKeysp= ace.java:819) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.SystemKeyspace.removeTruncationRecord(SystemK= eyspace.java:670) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore.invalidate(ColumnFamilyStor= e.java:550) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore.invalidate(ColumnFamilyStor= e.java:526) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.schema.LegacySchemaMigrator.lambda$unloadLegacyS= chemaTables$1(LegacySchemaMigrator.java:137) ~[apache-cassandra-3.11.1.jar:= 3.11.1] =09at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_131] =09at org.apache.cassandra.schema.LegacySchemaMigrator.unloadLegacySchemaTa= bles(LegacySchemaMigrator.java:137) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.schema.LegacySchemaMigrator.migrate(LegacySchema= Migrator.java:83) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.ja= va:256) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon= .java:600) [apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.jav= a:689) [apache-cassandra-3.11.1.jar:3.11.1] Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeExcept= ion: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/system/lo= cal-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e7-8e56-7= 1d63c551064.log =09at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.= 0_131] =09at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_1= 31] =09at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:= 400) ~[apache-cassandra-3.11.1.jar:3.11.1] =09... 11 common frames omitted Caused by: java.lang.RuntimeException: java.nio.file.AccessDeniedException:= /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_t= xn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590) ~[ap= ache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.java:5= 71) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.java:8= 5) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogRepl= icaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaSet.= java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:324)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285) ~[apa= che-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransact= ion.java:136) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(Lifec= ycleTransaction.java:529) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.format.big.BigTableWriter.(BigT= ableWriter.java:81) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.op= en(BigFormat.java:92) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWr= iter.java:102) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.create(Simpl= eSSTableMultiWriter.java:119) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.createS= STableMultiWriter(AbstractCompactionStrategy.java:587) ~[apache-cassandra-3= .11.1.jar:3.11.1] =09at org.apache.cassandra.db.compaction.CompactionStrategyManager.createSS= TableMultiWriter(CompactionStrategyManager.java:920) ~[apache-cassandra-3.1= 1.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore.createSSTableMultiWriter(Co= lumnFamilyStore.java:515) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memt= able.java:509) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable$FlushRunnable.(Memtable.java:4= 48) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable$FlushRunnable.(Memtable.java:4= 25) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:303) ~[= apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnF= amilyStore.java:1137) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore= .java:1102) ~[apache-cassandra-3.11.1.jar:3.11.1] =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1142) ~[na:1.8.0_131] =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:617) ~[na:1.8.0_131] =09at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocal= Deallocator$0(NamedThreadFactory.java:81) ~[apache-cassandra-3.11.1.jar:3.1= 1.1] =09at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] =09Suppressed: java.lang.RuntimeException: java.nio.file.AccessDeniedExcept= ion: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/= mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log =09=09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590) ~= [apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.jav= a:571) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.jav= a:85) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogR= eplicaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaS= et.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:3= 24) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogFile.abort(LogFile.java:257) = ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103)= ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:98) = ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LogTransaction.doAbort(LogTransa= ction.java:410) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransa= ctional.abort(Transactional.java:144) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.doAbort(Lif= ecycleTransaction.java:243) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransa= ctional.abort(Transactional.java:144) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(Colu= mnFamilyStore.java:1156) ~[apache-cassandra-3.11.1.jar:3.11.1] =09=09... 5 common frames omitted =09Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/= system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11= e7-8e56-71d63c551064.log =09=09at sun.nio.fs.UnixException.translateToIOException(UnixException.java= :84) =09=09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:1= 02) =09=09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:1= 07) =09=09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemPro= vider.java:214) =09=09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemPro= vider.java:434) =09=09at java.nio.file.Files.newOutputStream(Files.java:216) =09=09at java.nio.file.Files.write(Files.java:3351) =09=09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) =09=09... 21 common frames omitted Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/sys= tem/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e7-= 8e56-71d63c551064.log =09at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84= ) ~[na:1.8.0_131] =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)= ~[na:1.8.0_131] =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)= ~[na:1.8.0_131] =09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvid= er.java:214) ~[na:1.8.0_131] =09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvid= er.java:434) ~[na:1.8.0_131] =09at java.nio.file.Files.newOutputStream(Files.java:216) ~[na:1.8.0_131] =09at java.nio.file.Files.write(Files.java:3351) ~[na:1.8.0_131] =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) ~[ap= ache-cassandra-3.11.1.jar:3.11.1] =09... 27 common frames omitted {code} > node can't start, java.nio.file.AccessDeniedException to not existent fil= e > -------------------------------------------------------------------------= - > > Key: CASSANDRA-14161 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1416= 1 > Project: Cassandra > Issue Type: Bug > Environment: ubuntu +cassandra 3.11.1 > Reporter: Rogelio Trivi=C3=B1o Gonz=C3=A1lez > Fix For: 3.11.1 > > > this exception cuts the node starting: > {code} > java.lang.RuntimeException: java.nio.file.AccessDeniedException: /var/lib= /cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_= 6579bf70-f625-11e7-8e56-71d63c551064.log > {code} > the offending file doesn't exists: > {code} > ls -la /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860= b377/ > total 160 > drwxr-xr-x 3 cassandra cassandra 4096 ene 7 11:48 . > drwxr-xr-x 26 cassandra cassandra 4096 dic 18 07:30 .. > drwxr-xr-x 2 cassandra cassandra 4096 dic 18 07:30 backups > -rw-r--r-- 1 cassandra cassandra 43 ene 7 03:50 mc-100-big-Compressio= nInfo.db > -rw-r--r-- 1 cassandra cassandra 52 ene 7 03:50 mc-100-big-Data.db > -rw-r--r-- 1 cassandra cassandra 8 ene 7 03:50 mc-100-big-Digest.crc= 32 > -rw-r--r-- 1 cassandra cassandra 16 ene 7 03:50 mc-100-big-Filter.db > -rw-r--r-- 1 cassandra cassandra 9 ene 7 03:50 mc-100-big-Index.db > -rw-r--r-- 1 cassandra cassandra 4615 ene 7 03:50 mc-100-big-Statistics= .db > -rw-r--r-- 1 cassandra cassandra 59 ene 7 03:50 mc-100-big-Summary.db > -rw-r--r-- 1 cassandra cassandra 92 ene 7 03:50 mc-100-big-TOC.txt > -rw-r--r-- 1 cassandra cassandra 43 ene 7 11:48 mc-101-big-Compressio= nInfo.db > -rw-r--r-- 1 cassandra cassandra 50 ene 7 11:48 mc-101-big-Data.db > -rw-r--r-- 1 cassandra cassandra 10 ene 7 11:48 mc-101-big-Digest.crc= 32 > -rw-r--r-- 1 cassandra cassandra 16 ene 7 11:48 mc-101-big-Filter.db > -rw-r--r-- 1 cassandra cassandra 9 ene 7 11:48 mc-101-big-Index.db > -rw-r--r-- 1 cassandra cassandra 4615 ene 7 11:48 mc-101-big-Statistics= .db > -rw-r--r-- 1 cassandra cassandra 59 ene 7 11:48 mc-101-big-Summary.db > -rw-r--r-- 1 cassandra cassandra 92 ene 7 11:48 mc-101-big-TOC.txt > -rw-r--r-- 1 cassandra cassandra 51 ene 6 17:50 mc-98-big-Compression= Info.db > -rw-r--r-- 1 cassandra cassandra 5226 ene 6 17:50 mc-98-big-Data.db > -rw-r--r-- 1 cassandra cassandra 10 ene 6 17:50 mc-98-big-Digest.crc3= 2 > -rw-r--r-- 1 cassandra cassandra 16 ene 6 17:50 mc-98-big-Filter.db > -rw-r--r-- 1 cassandra cassandra 9 ene 6 17:50 mc-98-big-Index.db > -rw-r--r-- 1 cassandra cassandra 5654 ene 6 17:50 mc-98-big-Statistics.= db > -rw-r--r-- 1 cassandra cassandra 59 ene 6 17:50 mc-98-big-Summary.db > -rw-r--r-- 1 cassandra cassandra 92 ene 6 17:50 mc-98-big-TOC.txt > -rw-r--r-- 1 cassandra cassandra 43 ene 7 02:50 mc-99-big-Compression= Info.db > -rw-r--r-- 1 cassandra cassandra 50 ene 7 02:50 mc-99-big-Data.db > -rw-r--r-- 1 cassandra cassandra 10 ene 7 02:50 mc-99-big-Digest.crc3= 2 > -rw-r--r-- 1 cassandra cassandra 16 ene 7 02:50 mc-99-big-Filter.db > -rw-r--r-- 1 cassandra cassandra 9 ene 7 02:50 mc-99-big-Index.db > -rw-r--r-- 1 cassandra cassandra 4615 ene 7 02:50 mc-99-big-Statistics.= db > -rw-r--r-- 1 cassandra cassandra 59 ene 7 02:50 mc-99-big-Summary.db > -rw-r--r-- 1 cassandra cassandra 92 ene 7 02:50 mc-99-big-TOC.txt > {code} > permissions are ok, everything assigned to cassandra:cassandra > the system.log: > {code} > INFO [main] 2018-01-10 10:43:29,367 YamlConfigurationLoader.java:89 - Co= nfiguration location: file:/etc/cassandra/cassandra.yaml > INFO [main] 2018-01-10 10:43:29,887 Config.java:481 - Node configuration= :[allocate_tokens_for_keyspace=3Dnull; authenticator=3DAllowAllAuthenticato= r; authorizer=3DAllowAllAuthorizer; auto_bootstrap=3Dtrue; auto_snapshot=3D= true; back_pressure_enabled=3Dfalse; back_pressure_strategy=3Dnull; batch_s= ize_fail_threshold_in_kb=3D500; batch_size_warn_threshold_in_kb=3D50; batch= log_replay_throttle_in_kb=3D1024; broadcast_address=3Dnull; broadcast_rpc_a= ddress=3Dnull; buffer_pool_use_heap_if_exhausted=3Dtrue; cas_contention_tim= eout_in_ms=3D10000; cdc_enabled=3Dfalse; cdc_free_space_check_interval_ms= =3D250; cdc_raw_directory=3Dnull; cdc_total_space_in_mb=3D0; client_encrypt= ion_options=3D; cluster_name=3DPAP Cluster; column_index_cache_si= ze_in_kb=3D2; column_index_size_in_kb=3D64; commit_failure_policy=3Dstop; c= ommitlog_compression=3Dnull; commitlog_directory=3D/var/lib/cassandra/commi= tlog; commitlog_max_compression_buffers_in_pool=3D3; commitlog_periodic_que= ue_size=3D-1; commitlog_segment_size_in_mb=3D32; commitlog_sync=3Dperiodic;= commitlog_sync_batch_window_in_ms=3DNaN; commitlog_sync_period_in_ms=3D100= 00; commitlog_total_space_in_mb=3Dnull; compaction_large_partition_warning_= threshold_mb=3D100; compaction_throughput_mb_per_sec=3D16; concurrent_compa= ctors=3Dnull; concurrent_counter_writes=3D32; concurrent_materialized_view_= writes=3D32; concurrent_reads=3D32; concurrent_replicates=3Dnull; concurren= t_writes=3D32; counter_cache_keys_to_save=3D2147483647; counter_cache_save_= period=3D7200; counter_cache_size_in_mb=3Dnull; counter_write_request_timeo= ut_in_ms=3D50000; credentials_cache_max_entries=3D1000; credentials_update_= interval_in_ms=3D-1; credentials_validity_in_ms=3D2000; cross_node_timeout= =3Dfalse; data_file_directories=3D[Ljava.lang.String;@79ca92b9; disk_access= _mode=3Dauto; disk_failure_policy=3Dstop; disk_optimization_estimate_percen= tile=3D0.95; disk_optimization_page_cross_chance=3D0.1; disk_optimization_s= trategy=3Dssd; dynamic_snitch=3Dtrue; dynamic_snitch_badness_threshold=3D0.= 1; dynamic_snitch_reset_interval_in_ms=3D600000; dynamic_snitch_update_inte= rval_in_ms=3D100; enable_scripted_user_defined_functions=3Dfalse; enable_us= er_defined_functions=3Dfalse; enable_user_defined_functions_threads=3Dtrue;= encryption_options=3Dnull; endpoint_snitch=3DGossipingPropertyFileSnitch; = file_cache_size_in_mb=3Dnull; gc_log_threshold_in_ms=3D200; gc_warn_thresho= ld_in_ms=3D1000; hinted_handoff_disabled_datacenters=3D[]; hinted_handoff_e= nabled=3Dtrue; hinted_handoff_throttle_in_kb=3D1024; hints_compression=3Dnu= ll; hints_directory=3Dnull; hints_flush_period_in_ms=3D10000; incremental_b= ackups=3Dfalse; index_interval=3Dnull; index_summary_capacity_in_mb=3Dnull;= index_summary_resize_interval_in_minutes=3D60; initial_token=3Dnull; inter= _dc_stream_throughput_outbound_megabits_per_sec=3D200; inter_dc_tcp_nodelay= =3Dfalse; internode_authenticator=3Dnull; internode_compression=3Dall; inte= rnode_recv_buff_size_in_bytes=3D0; internode_send_buff_size_in_bytes=3D0; k= ey_cache_keys_to_save=3D2147483647; key_cache_save_period=3D14400; key_cach= e_size_in_mb=3Dnull; listen_address=3D192.121.166.40; listen_interface=3Dnu= ll; listen_interface_prefer_ipv6=3Dfalse; listen_on_broadcast_address=3Dfal= se; max_hint_window_in_ms=3D10800000; max_hints_delivery_threads=3D2; max_h= ints_file_size_in_mb=3D128; max_mutation_size_in_kb=3Dnull; max_streaming_r= etries=3D3; max_value_size_in_mb=3D256; memtable_allocation_type=3Dheap_buf= fers; memtable_cleanup_threshold=3Dnull; memtable_flush_writers=3D0; memtab= le_heap_space_in_mb=3Dnull; memtable_offheap_space_in_mb=3Dnull; min_free_s= pace_per_drive_in_mb=3D50; native_transport_max_concurrent_connections=3D-1= ; native_transport_max_concurrent_connections_per_ip=3D-1; native_transport= _max_frame_size_in_mb=3D256; native_transport_max_threads=3D128; native_tra= nsport_port=3D9042; native_transport_port_ssl=3Dnull; num_tokens=3D256; otc= _backlog_expiration_interval_ms=3D200; otc_coalescing_enough_coalesced_mess= ages=3D8; otc_coalescing_strategy=3DDISABLED; otc_coalescing_window_us=3D20= 0; partitioner=3Dorg.apache.cassandra.dht.Murmur3Partitioner; permissions_c= ache_max_entries=3D1000; permissions_update_interval_in_ms=3D-1; permission= s_validity_in_ms=3D2000; phi_convict_threshold=3D8.0; prepared_statements_c= ache_size_mb=3Dnull; range_request_timeout_in_ms=3D100000; read_request_tim= eout_in_ms=3D100000; request_scheduler=3Dorg.apache.cassandra.scheduler.NoS= cheduler; request_scheduler_id=3Dnull; request_scheduler_options=3Dnull; re= quest_timeout_in_ms=3D100000; role_manager=3DCassandraRoleManager; roles_ca= che_max_entries=3D1000; roles_update_interval_in_ms=3D-1; roles_validity_in= _ms=3D2000; row_cache_class_name=3Dorg.apache.cassandra.cache.OHCProvider; = row_cache_keys_to_save=3D2147483647; row_cache_save_period=3D0; row_cache_s= ize_in_mb=3D0; rpc_address=3Dlocalhost; rpc_interface=3Dnull; rpc_interface= _prefer_ipv6=3Dfalse; rpc_keepalive=3Dtrue; rpc_listen_backlog=3D50; rpc_ma= x_threads=3D2147483647; rpc_min_threads=3D16; rpc_port=3D9160; rpc_recv_buf= f_size_in_bytes=3Dnull; rpc_send_buff_size_in_bytes=3Dnull; rpc_server_type= =3Dsync; saved_caches_directory=3D/var/lib/cassandra/saved_caches; seed_pro= vider=3Dorg.apache.cassandra.locator.SimpleSeedProvider{seeds=3D63.142.255.= 107,37.247.116.68}; server_encryption_options=3D; slow_query_log_= timeout_in_ms=3D500; snapshot_before_compaction=3Dfalse; ssl_storage_port= =3D7071; sstable_preemptive_open_interval_in_mb=3D50; start_native_transpor= t=3Dtrue; start_rpc=3Dfalse; storage_port=3D7070; stream_throughput_outboun= d_megabits_per_sec=3D200; streaming_keep_alive_period_in_secs=3D300; stream= ing_socket_timeout_in_ms=3D86400000; thrift_framed_transport_size_in_mb=3D1= 5; thrift_max_message_length_in_mb=3D16; thrift_prepared_statements_cache_s= ize_mb=3Dnull; tombstone_failure_threshold=3D100000; tombstone_warn_thresho= ld=3D1000; tracetype_query_ttl=3D86400; tracetype_repair_ttl=3D604800; tran= sparent_data_encryption_options=3Dorg.apache.cassandra.config.TransparentDa= taEncryptionOptions@1460a8c0; trickle_fsync=3Dfalse; trickle_fsync_interval= _in_kb=3D10240; truncate_request_timeout_in_ms=3D600000; unlogged_batch_acr= oss_partitions_warn_threshold=3D10; user_defined_function_fail_timeout=3D15= 00; user_defined_function_warn_timeout=3D500; user_function_timeout_policy= =3Ddie; windows_timer_interval=3D1; write_request_timeout_in_ms=3D50000] > INFO [main] 2018-01-10 10:43:29,888 DatabaseDescriptor.java:367 - DiskAc= cessMode 'auto' determined to be mmap, indexAccessMode is mmap > INFO [main] 2018-01-10 10:43:29,888 DatabaseDescriptor.java:421 - Global= memtable on-heap threshold is enabled at 374MB > INFO [main] 2018-01-10 10:43:29,888 DatabaseDescriptor.java:425 - Global= memtable off-heap threshold is enabled at 374MB > WARN [main] 2018-01-10 10:43:29,897 DatabaseDescriptor.java:470 - Small = commitlog volume detected at /var/lib/cassandra/commitlog; setting commitlo= g_total_space_in_mb to 7680. You can override this in cassandra.yaml > WARN [main] 2018-01-10 10:43:29,899 DatabaseDescriptor.java:497 - Small = cdc volume detected at /var/lib/cassandra/cdc_raw; setting cdc_total_space_= in_mb to 3840. You can override this in cassandra.yaml > WARN [main] 2018-01-10 10:43:30,022 DatabaseDescriptor.java:546 - Only 1= 8.837GiB free across all data volumes. Consider adding more capacity to you= r cluster or removing obsolete snapshots > INFO [main] 2018-01-10 10:43:30,066 RateBasedBackPressure.java:123 - Ini= tialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window = size: 50000. > INFO [main] 2018-01-10 10:43:30,066 DatabaseDescriptor.java:725 - Back-p= ressure is disabled with strategy null. > INFO [main] 2018-01-10 10:43:30,250 GossipingPropertyFileSnitch.java:64 = - Loaded cassandra-topology.properties for compatibility > INFO [main] 2018-01-10 10:43:30,329 JMXServerUtils.java:249 - Configured= JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrm= i > INFO [main] 2018-01-10 10:43:30,335 CassandraDaemon.java:471 - Hostname:= uk1.plugandplay.click > INFO [main] 2018-01-10 10:43:30,335 CassandraDaemon.java:478 - JVM vendo= r/version: Java HotSpot(TM) 64-Bit Server VM/1.8.0_131 > INFO [main] 2018-01-10 10:43:30,336 CassandraDaemon.java:479 - Heap size= : 1.463GiB/1.463GiB > INFO [main] 2018-01-10 10:43:30,336 CassandraDaemon.java:484 - Code Cach= e Non-heap memory: init =3D 2555904(2496K) used =3D 4545344(4438K) committe= d =3D 4587520(4480K) max =3D 251658240(245760K) > INFO [main] 2018-01-10 10:43:30,337 CassandraDaemon.java:484 - Metaspace= Non-heap memory: init =3D 0(0K) used =3D 19068664(18621K) committed =3D 19= 791872(19328K) max =3D -1(-1K) > INFO [main] 2018-01-10 10:43:30,337 CassandraDaemon.java:484 - Compresse= d Class Space Non-heap memory: init =3D 0(0K) used =3D 2353456(2298K) commi= tted =3D 2490368(2432K) max =3D 1073741824(1048576K) > INFO [main] 2018-01-10 10:43:30,337 CassandraDaemon.java:484 - Par Eden = Space Heap memory: init =3D 322174976(314624K) used =3D 109586656(107018K) = committed =3D 322174976(314624K) max =3D 322174976(314624K) > INFO [main] 2018-01-10 10:43:30,338 CassandraDaemon.java:484 - Par Survi= vor Space Heap memory: init =3D 40239104(39296K) used =3D 0(0K) committed = =3D 40239104(39296K) max =3D 40239104(39296K) > INFO [main] 2018-01-10 10:43:30,338 CassandraDaemon.java:484 - CMS Old G= en Heap memory: init =3D 1207959552(1179648K) used =3D 0(0K) committed =3D = 1207959552(1179648K) max =3D 1207959552(1179648K) > INFO [main] 2018-01-10 10:43:30,338 CassandraDaemon.java:486 - Classpath= : /etc/cassandra:/usr/share/cassandra/lib/HdrHistogram-2.1.9.jar:/usr/share= /cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/= share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-5.= 0.4.jar:/usr/share/cassandra/lib/caffeine-2.2.6.jar:/usr/share/cassandra/li= b/cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/commons-c= li-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.9.jar:/usr/share/cassan= dra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.ja= r:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/= concurrent-trees-2.4.0.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap= -lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassan= dra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-18.0.jar:/usr/share/ca= ssandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/hppc-0.5.4.ja= r:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/= lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/u= sr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.= 3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassand= ra/lib/jctools-core-1.2.1.jar:/usr/share/cassandra/lib/jflex-1.6.0.jar:/usr= /share/cassandra/lib/jna-4.2.2.jar:/usr/share/cassandra/lib/joda-time-2.4.j= ar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/js= tackjunit-0.0.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share= /cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-= classic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/shar= e/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.j= ar:/usr/share/cassandra/lib/metrics-jvm-3.1.0.jar:/usr/share/cassandra/lib/= metrics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.44.Final.j= ar:/usr/share/cassandra/lib/ohc-core-0.4.4.jar:/usr/share/cassandra/lib/ohc= -core-j8-0.4.4.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.3.jar:= /usr/share/cassandra/lib/reporter-config3-3.0.3.jar:/usr/share/cassandra/li= b/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/c= assandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.= 7.jar:/usr/share/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/usr/share/= cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7= .jar:/usr/share/cassandra/apache-cassandra-3.11.1.jar:/usr/share/cassandra/= apache-cassandra-thrift-3.11.1.jar:/usr/share/cassandra/apache-cassandra.ja= r:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar > INFO [main] 2018-01-10 10:43:30,339 CassandraDaemon.java:488 - JVM Argum= ents: [-Xloggc:/var/log/cassandra/gc.log, -ea, -XX:+UseThreadPriorities, -X= X:ThreadPriorityPolicy=3D42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX= :StringTableSize=3D1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX= :+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava= .net.preferIPv4Stack=3Dtrue, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX= :+CMSParallelRemarkEnabled, -XX:SurvivorRatio=3D8, -XX:MaxTenuringThreshold= =3D1, -XX:CMSInitiatingOccupancyFraction=3D75, -XX:+UseCMSInitiatingOccupan= cyOnly, -XX:CMSWaitDuration=3D10000, -XX:+CMSParallelInitialMarkEnabled, -X= X:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDe= tails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistri= bution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX= :+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=3D10, -XX:GCLogFileSize=3D10= M, -Xms1536M, -Xmx1536M, -Xmn384M, -XX:+UseCondCardMark, -XX:CompileCommand= File=3D/etc/cassandra/hotspot_compiler, -javaagent:/usr/share/cassandra/lib= /jamm-0.3.0.jar, -Dcassandra.jmx.local.port=3D7199, -Dcom.sun.management.jm= xremote.authenticate=3Dfalse, -Dcom.sun.management.jmxremote.password.file= =3D/etc/cassandra/jmxremote.password, -Djava.library.path=3D/usr/share/cass= andra/lib/sigar-bin, -Dlogback.configurationFile=3Dlogback.xml, -Dcassandra= .logdir=3D/var/log/cassandra, -Dcassandra.storagedir=3D/var/lib/cassandra, = -Dcassandra-pidfile=3D/var/run/cassandra/cassandra.pid, -XX:HeapDumpPath=3D= /var/lib/cassandra/java_1515602606.hprof, -XX:ErrorFile=3D/var/lib/cassandr= a/hs_err_1515602606.log] > INFO [main] 2018-01-10 10:43:30,598 NativeLibrary.java:174 - JNA mlockal= l successful > WARN [main] 2018-01-10 10:43:30,599 StartupChecks.java:133 - jemalloc sh= ared library could not be preloaded to speed up memory allocations > WARN [main] 2018-01-10 10:43:30,599 StartupChecks.java:166 - JMX is not = enabled to receive remote connections. Please see cassandra-env.sh for more= info. > INFO [main] 2018-01-10 10:43:30,601 SigarLibrary.java:44 - Initializing = SIGAR library > INFO [main] 2018-01-10 10:43:30,614 SigarLibrary.java:180 - Checked OS s= ettings and found them configured for optimal performance. > WARN [main] 2018-01-10 10:43:30,619 StartupChecks.java:271 - Maximum num= ber of memory map areas per process (vm.max_map_count) 65530 is too low, re= commended value: 1048575, you can change it with sysctl. > INFO [main] 2018-01-10 10:43:30,975 QueryProcessor.java:116 - Initialize= d prepared statement caches with 10 MB (native) and 10 MB (Thrift) > INFO [main] 2018-01-10 10:43:31,541 ColumnFamilyStore.java:408 - Initial= izing system.IndexInfo > INFO [SSTableBatchOpen:1] 2018-01-10 10:43:32,405 BufferPool.java:230 - = Global buffer pool is enabled, when pool is exhausted (max is 374.000MiB) i= t will allocate on heap > INFO [main] 2018-01-10 10:43:32,653 CacheService.java:112 - Initializing= key cache with capacity of 74 MBs. > INFO [main] 2018-01-10 10:43:32,662 CacheService.java:134 - Initializing= row cache with capacity of 0 MBs > INFO [main] 2018-01-10 10:43:32,664 CacheService.java:163 - Initializing= counter cache with capacity of 37 MBs > INFO [main] 2018-01-10 10:43:32,666 CacheService.java:174 - Scheduling c= ounter cache save to every 7200 seconds (going to save all keys). > INFO [main] 2018-01-10 10:43:32,720 ColumnFamilyStore.java:408 - Initial= izing system.batches > INFO [main] 2018-01-10 10:43:32,747 ColumnFamilyStore.java:408 - Initial= izing system.paxos > INFO [main] 2018-01-10 10:43:32,763 ColumnFamilyStore.java:408 - Initial= izing system.local > INFO [main] 2018-01-10 10:43:32,791 ColumnFamilyStore.java:408 - Initial= izing system.peers > INFO [main] 2018-01-10 10:43:32,814 ColumnFamilyStore.java:408 - Initial= izing system.peer_events > INFO [main] 2018-01-10 10:43:32,821 ColumnFamilyStore.java:408 - Initial= izing system.range_xfers > INFO [main] 2018-01-10 10:43:32,830 ColumnFamilyStore.java:408 - Initial= izing system.compaction_history > INFO [main] 2018-01-10 10:43:32,852 ColumnFamilyStore.java:408 - Initial= izing system.sstable_activity > INFO [main] 2018-01-10 10:43:32,977 ColumnFamilyStore.java:408 - Initial= izing system.size_estimates > INFO [main] 2018-01-10 10:43:32,992 ColumnFamilyStore.java:408 - Initial= izing system.available_ranges > INFO [main] 2018-01-10 10:43:33,002 ColumnFamilyStore.java:408 - Initial= izing system.transferred_ranges > INFO [main] 2018-01-10 10:43:33,008 ColumnFamilyStore.java:408 - Initial= izing system.views_builds_in_progress > INFO [main] 2018-01-10 10:43:33,014 ColumnFamilyStore.java:408 - Initial= izing system.built_views > INFO [main] 2018-01-10 10:43:33,021 ColumnFamilyStore.java:408 - Initial= izing system.hints > INFO [main] 2018-01-10 10:43:33,028 ColumnFamilyStore.java:408 - Initial= izing system.batchlog > INFO [main] 2018-01-10 10:43:33,036 ColumnFamilyStore.java:408 - Initial= izing system.prepared_statements > INFO [main] 2018-01-10 10:43:33,049 ColumnFamilyStore.java:408 - Initial= izing system.schema_keyspaces > INFO [main] 2018-01-10 10:43:33,055 ColumnFamilyStore.java:408 - Initial= izing system.schema_columnfamilies > INFO [main] 2018-01-10 10:43:33,062 ColumnFamilyStore.java:408 - Initial= izing system.schema_columns > INFO [main] 2018-01-10 10:43:33,068 ColumnFamilyStore.java:408 - Initial= izing system.schema_triggers > INFO [main] 2018-01-10 10:43:33,074 ColumnFamilyStore.java:408 - Initial= izing system.schema_usertypes > INFO [main] 2018-01-10 10:43:33,081 ColumnFamilyStore.java:408 - Initial= izing system.schema_functions > INFO [main] 2018-01-10 10:43:33,088 ColumnFamilyStore.java:408 - Initial= izing system.schema_aggregates > INFO [main] 2018-01-10 10:43:33,090 ViewManager.java:137 - Not submittin= g build tasks for views in keyspace system as storage service is not initia= lized > INFO [main] 2018-01-10 10:43:33,250 ApproximateTime.java:44 - Scheduling= approximate time-check task with a precision of 10 milliseconds > INFO [main] 2018-01-10 10:43:33,307 ColumnFamilyStore.java:408 - Initial= izing system_schema.keyspaces > INFO [main] 2018-01-10 10:43:33,322 ColumnFamilyStore.java:408 - Initial= izing system_schema.tables > INFO [main] 2018-01-10 10:43:33,336 ColumnFamilyStore.java:408 - Initial= izing system_schema.columns > INFO [main] 2018-01-10 10:43:33,347 ColumnFamilyStore.java:408 - Initial= izing system_schema.triggers > INFO [main] 2018-01-10 10:43:33,353 ColumnFamilyStore.java:408 - Initial= izing system_schema.dropped_columns > INFO [main] 2018-01-10 10:43:33,359 ColumnFamilyStore.java:408 - Initial= izing system_schema.views > INFO [main] 2018-01-10 10:43:33,364 ColumnFamilyStore.java:408 - Initial= izing system_schema.types > INFO [main] 2018-01-10 10:43:33,370 ColumnFamilyStore.java:408 - Initial= izing system_schema.functions > INFO [main] 2018-01-10 10:43:33,376 ColumnFamilyStore.java:408 - Initial= izing system_schema.aggregates > INFO [main] 2018-01-10 10:43:33,382 ColumnFamilyStore.java:408 - Initial= izing system_schema.indexes > INFO [main] 2018-01-10 10:43:33,388 ViewManager.java:137 - Not submittin= g build tasks for views in keyspace system_schema as storage service is not= initialized > ERROR [MemtableFlushWriter:1] 2018-01-10 10:43:33,507 LogTransaction.java= :273 - Transaction log [mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.l= og in /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377= ] indicates txn was not completed, trying to abort it now > ERROR [MemtableFlushWriter:1] 2018-01-10 10:43:33,512 LogTransaction.java= :276 - Failed to abort transaction log [mc_txn_flush_6579bf70-f625-11e7-8e5= 6-71d63c551064.log in /var/lib/cassandra/data/system/local-7ad54392bcdd35a6= 84174e047860b377] > java.lang.RuntimeException: java.nio.file.AccessDeniedException: /var/lib= /cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_= 6579bf70-f625-11e7-8e56-71d63c551064.log > =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590) ~[= apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.java= :571) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.java= :85) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogRe= plicaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) = ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) = ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaSe= t.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:32= 4) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogFile.abort(LogFile.java:257) ~= [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) = ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) = ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:98) ~= [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogTransaction$TransactionTidier.= run(LogTransaction.java:274) [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogTransaction$TransactionTidier.= tidy(LogTransaction.java:258) [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.j= ava:321) [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.= java:199) [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:1= 19) [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogTransaction.complete(LogTransa= ction.java:393) [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogTransaction.doAbort(LogTransac= tion.java:410) [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransac= tional.abort(Transactional.java:144) [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.doAbort(Life= cycleTransaction.java:243) [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransac= tional.abort(Transactional.java:144) [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(Colum= nFamilyStore.java:1156) [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilySto= re.java:1102) [apache-cassandra-3.11.1.jar:3.11.1] > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1142) [na:1.8.0_131] > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:617) [na:1.8.0_131] > =09at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLoc= alDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.= 11.1] > =09at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] > Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/s= ystem/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e= 7-8e56-71d63c551064.log > =09at sun.nio.fs.UnixException.translateToIOException(UnixException.java:= 84) ~[na:1.8.0_131] > =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:10= 2) ~[na:1.8.0_131] > =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:10= 7) ~[na:1.8.0_131] > =09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProv= ider.java:214) ~[na:1.8.0_131] > =09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProv= ider.java:434) ~[na:1.8.0_131] > =09at java.nio.file.Files.newOutputStream(Files.java:216) ~[na:1.8.0_131] > =09at java.nio.file.Files.write(Files.java:3351) ~[na:1.8.0_131] > =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) ~[= apache-cassandra-3.11.1.jar:3.11.1] > =09... 27 common frames omitted > ERROR [MemtableFlushWriter:1] 2018-01-10 10:43:33,515 LogTransaction.java= :223 - Unable to delete /var/lib/cassandra/data/system/local-7ad54392bcdd35= a684174e047860b377/mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log as= it does not exist, see debug log file for stack trace > ERROR [MemtablePostFlush:1] 2018-01-10 10:43:33,520 CassandraDaemon.java:= 228 - Exception in thread Thread[MemtablePostFlush:1,5,main] > java.lang.RuntimeException: java.nio.file.AccessDeniedException: /var/lib= /cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_= 6579bf70-f625-11e7-8e56-71d63c551064.log > =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590) ~[= apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.java= :571) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.java= :85) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogRe= plicaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) = ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) = ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaSe= t.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:32= 4) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285) ~[a= pache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransa= ction.java:136) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(Lif= ecycleTransaction.java:529) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.io.sstable.format.big.BigTableWriter.(Bi= gTableWriter.java:81) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.= open(BigFormat.java:92) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTable= Writer.java:102) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.create(Sim= pleSSTableMultiWriter.java:119) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.creat= eSSTableMultiWriter(AbstractCompactionStrategy.java:587) ~[apache-cassandra= -3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.compaction.CompactionStrategyManager.create= SSTableMultiWriter(CompactionStrategyManager.java:920) ~[apache-cassandra-3= .11.1.jar:3.11.1] > =09at org.apache.cassandra.db.ColumnFamilyStore.createSSTableMultiWriter(= ColumnFamilyStore.java:515) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Me= mtable.java:509) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.Memtable$FlushRunnable.(Memtable.java= :448) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.Memtable$FlushRunnable.(Memtable.java= :425) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:303) = ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(Colum= nFamilyStore.java:1137) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilySto= re.java:1102) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1142) ~[na:1.8.0_131] > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:617) [na:1.8.0_131] > =09at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLoc= alDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.= 11.1] > =09at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] > =09Suppressed: java.lang.RuntimeException: java.nio.file.AccessDeniedExce= ption: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b37= 7/mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log > =09=09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590)= ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.j= ava:571) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.j= ava:85) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(Lo= gReplicaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:11= 3) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:10= 3) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplic= aSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java= :324) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.db.lifecycle.LogFile.abort(LogFile.java:257= ) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:11= 3) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:10= 3) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:98= ) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.db.lifecycle.LogTransaction.doAbort(LogTran= saction.java:410) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTran= sactional.abort(Transactional.java:144) ~[apache-cassandra-3.11.1.jar:3.11.= 1] > =09=09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.doAbort(L= ifecycleTransaction.java:243) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTran= sactional.abort(Transactional.java:144) ~[apache-cassandra-3.11.1.jar:3.11.= 1] > =09=09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(Co= lumnFamilyStore.java:1156) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09... 5 common frames omitted > =09Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/dat= a/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-= 11e7-8e56-71d63c551064.log > =09=09at sun.nio.fs.UnixException.translateToIOException(UnixException.ja= va:84) > =09=09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java= :102) > =09=09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java= :107) > =09=09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemP= rovider.java:214) > =09=09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemP= rovider.java:434) > =09=09at java.nio.file.Files.newOutputStream(Files.java:216) > =09=09at java.nio.file.Files.write(Files.java:3351) > =09=09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) > =09=09... 21 common frames omitted > Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/s= ystem/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e= 7-8e56-71d63c551064.log > =09at sun.nio.fs.UnixException.translateToIOException(UnixException.java:= 84) ~[na:1.8.0_131] > =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:10= 2) ~[na:1.8.0_131] > =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:10= 7) ~[na:1.8.0_131] > =09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProv= ider.java:214) ~[na:1.8.0_131] > =09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProv= ider.java:434) ~[na:1.8.0_131] > =09at java.nio.file.Files.newOutputStream(Files.java:216) ~[na:1.8.0_131] > =09at java.nio.file.Files.write(Files.java:3351) ~[na:1.8.0_131] > =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) ~[= apache-cassandra-3.11.1.jar:3.11.1] > =09... 27 common frames omitted > ERROR [main] 2018-01-10 10:43:33,522 CassandraDaemon.java:706 - Exception= encountered during startup > java.lang.RuntimeException: java.util.concurrent.ExecutionException: java= .lang.RuntimeException: java.nio.file.AccessDeniedException: /var/lib/cassa= ndra/data/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf= 70-f625-11e7-8e56-71d63c551064.log > =09at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.jav= a:404) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.SystemKeyspace.forceBlockingFlush(SystemKey= space.java:819) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.SystemKeyspace.removeTruncationRecord(Syste= mKeyspace.java:670) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.ColumnFamilyStore.invalidate(ColumnFamilySt= ore.java:550) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.ColumnFamilyStore.invalidate(ColumnFamilySt= ore.java:526) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.schema.LegacySchemaMigrator.lambda$unloadLegac= ySchemaTables$1(LegacySchemaMigrator.java:137) ~[apache-cassandra-3.11.1.ja= r:3.11.1] > =09at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_131] > =09at org.apache.cassandra.schema.LegacySchemaMigrator.unloadLegacySchema= Tables(LegacySchemaMigrator.java:137) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.schema.LegacySchemaMigrator.migrate(LegacySche= maMigrator.java:83) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.= java:256) [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaem= on.java:600) [apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.j= ava:689) [apache-cassandra-3.11.1.jar:3.11.1] > Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeExce= ption: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/system/= local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e7-8e56= -71d63c551064.log > =09at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.= 8.0_131] > =09at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0= _131] > =09at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.jav= a:400) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09... 11 common frames omitted > Caused by: java.lang.RuntimeException: java.nio.file.AccessDeniedExceptio= n: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/mc= _txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log > =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590) ~[= apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.java= :571) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.java= :85) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(LogRe= plicaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:113) = ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:103) = ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplicaSe= t.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java:32= 4) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285) ~[a= pache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransa= ction.java:136) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(Lif= ecycleTransaction.java:529) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.io.sstable.format.big.BigTableWriter.(Bi= gTableWriter.java:81) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.= open(BigFormat.java:92) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTable= Writer.java:102) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.create(Sim= pleSSTableMultiWriter.java:119) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.creat= eSSTableMultiWriter(AbstractCompactionStrategy.java:587) ~[apache-cassandra= -3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.compaction.CompactionStrategyManager.create= SSTableMultiWriter(CompactionStrategyManager.java:920) ~[apache-cassandra-3= .11.1.jar:3.11.1] > =09at org.apache.cassandra.db.ColumnFamilyStore.createSSTableMultiWriter(= ColumnFamilyStore.java:515) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Me= mtable.java:509) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.Memtable$FlushRunnable.(Memtable.java= :448) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.Memtable$FlushRunnable.(Memtable.java= :425) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:303) = ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(Colum= nFamilyStore.java:1137) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilySto= re.java:1102) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1142) ~[na:1.8.0_131] > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:617) ~[na:1.8.0_131] > =09at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLoc= alDeallocator$0(NamedThreadFactory.java:81) ~[apache-cassandra-3.11.1.jar:3= .11.1] > =09at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] > =09Suppressed: java.lang.RuntimeException: java.nio.file.AccessDeniedExce= ption: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b37= 7/mc_txn_flush_6579bf70-f625-11e7-8e56-71d63c551064.log > =09=09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:590)= ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.io.util.FileUtils.appendAndSync(FileUtils.j= ava:571) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.db.lifecycle.LogReplica.append(LogReplica.j= ava:85) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.db.lifecycle.LogReplicaSet.lambda$null$5(Lo= gReplicaSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:11= 3) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:10= 3) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.db.lifecycle.LogReplicaSet.append(LogReplic= aSet.java:210) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.db.lifecycle.LogFile.addRecord(LogFile.java= :324) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.db.lifecycle.LogFile.abort(LogFile.java:257= ) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:11= 3) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:10= 3) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.utils.Throwables.perform(Throwables.java:98= ) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.db.lifecycle.LogTransaction.doAbort(LogTran= saction.java:410) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTran= sactional.abort(Transactional.java:144) ~[apache-cassandra-3.11.1.jar:3.11.= 1] > =09=09at org.apache.cassandra.db.lifecycle.LifecycleTransaction.doAbort(L= ifecycleTransaction.java:243) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09at org.apache.cassandra.utils.concurrent.Transactional$AbstractTran= sactional.abort(Transactional.java:144) ~[apache-cassandra-3.11.1.jar:3.11.= 1] > =09=09at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(Co= lumnFamilyStore.java:1156) ~[apache-cassandra-3.11.1.jar:3.11.1] > =09=09... 5 common frames omitted > =09Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/dat= a/system/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-= 11e7-8e56-71d63c551064.log > =09=09at sun.nio.fs.UnixException.translateToIOException(UnixException.ja= va:84) > =09=09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java= :102) > =09=09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java= :107) > =09=09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemP= rovider.java:214) > =09=09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemP= rovider.java:434) > =09=09at java.nio.file.Files.newOutputStream(Files.java:216) > =09=09at java.nio.file.Files.write(Files.java:3351) > =09=09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) > =09=09... 21 common frames omitted > Caused by: java.nio.file.AccessDeniedException: /var/lib/cassandra/data/s= ystem/local-7ad54392bcdd35a684174e047860b377/mc_txn_flush_6579bf70-f625-11e= 7-8e56-71d63c551064.log > =09at sun.nio.fs.UnixException.translateToIOException(UnixException.java:= 84) ~[na:1.8.0_131] > =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:10= 2) ~[na:1.8.0_131] > =09at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:10= 7) ~[na:1.8.0_131] > =09at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProv= ider.java:214) ~[na:1.8.0_131] > =09at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProv= ider.java:434) ~[na:1.8.0_131] > =09at java.nio.file.Files.newOutputStream(Files.java:216) ~[na:1.8.0_131] > =09at java.nio.file.Files.write(Files.java:3351) ~[na:1.8.0_131] > =09at org.apache.cassandra.io.util.FileUtils.write(FileUtils.java:583) ~[= apache-cassandra-3.11.1.jar:3.11.1] > =09... 27 common frames omitted > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org For additional commands, e-mail: commits-help@cassandra.apache.org