Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 15FA618EDF for ; Mon, 20 Jul 2015 13:32:05 +0000 (UTC) Received: (qmail 9944 invoked by uid 500); 20 Jul 2015 13:32:05 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 9911 invoked by uid 500); 20 Jul 2015 13:32:04 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 9897 invoked by uid 99); 20 Jul 2015 13:32:04 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2015 13:32:04 +0000 Date: Mon, 20 Jul 2015 13:32:04 +0000 (UTC) From: "Philip Thompson (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-9846) CassandraDaemon fails to start due to FileNotFoundException MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-9846?page=3Dcom.atlas= sian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D= 14633603#comment-14633603 ]=20 Philip Thompson commented on CASSANDRA-9846: -------------------------------------------- [~JoshuaMcKenzie], what do you think? > CassandraDaemon fails to start due to FileNotFoundException > ----------------------------------------------------------- > > Key: CASSANDRA-9846 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9846 > Project: Cassandra > Issue Type: Bug > Environment: Windows 7 64bit, 4-core, 24GB memory. Running single= cluster. > Java 1.8.0_45, python 2.7.10 > Reporter: Will Zhang > Attachments: system.2.1.8.log > > > Cassandra 2.1.8 > Server fails to start after seeing a `FileNotFoundException` in the logs. > Before upgrading to `2.1.8`, I was running `2.1.5`. I ran the following b= efore shutting down `2.1.5`: > nodetool flush > nodetool drain > notetool repair > When I tred to start 2.1.8 after a reboot, the following error is observe= d in the system.log file. Server would not start after reach this point. I = also attached the full error log which shows two start attempts.=20 > When I reverted back to 2.1.5, server started just fine, with no error in= log. > I saw the following issue reported but that was for 2.0.9. However, I hav= e not seen this error in 2.1.3, 2.1.4 or 2.1.5.=20 > https://issues.apache.org/jira/browse/CASSANDRA-7780=20 > Error log: > {code:java} > INFO [main] 2015-07-18 21:30:05,582 YamlConfigurationLoader.java:135 - N= ode configuration:[authenticator=3DAllowAllAuthenticator; authorizer=3DAllo= wAllAuthorizer; auto_snapshot=3Dtrue; batch_size_warn_threshold_in_kb=3D5; = batchlog_replay_throttle_in_kb=3D1024; cas_contention_timeout_in_ms=3D5000;= client_encryption_options=3D; cluster_name=3Dzhangwe; column_ind= ex_size_in_kb=3D64; commit_failure_policy=3Dstop; commitlog_directory=3Dz:/= cassandra/commitlog; commitlog_segment_size_in_mb=3D32; commitlog_sync=3Dpe= riodic; commitlog_sync_period_in_ms=3D10000; compaction_throughput_mb_per_s= ec=3D16; concurrent_counter_writes=3D32; concurrent_reads=3D32; concurrent_= writes=3D32; counter_cache_save_period=3D7200; counter_cache_size_in_mb=3Dn= ull; counter_write_request_timeout_in_ms=3D40000; cross_node_timeout=3Dfals= e; data_file_directories=3D[z:/cassandra/data]; disk_failure_policy=3Dstop;= dynamic_snitch_badness_threshold=3D0.1; dynamic_snitch_reset_interval_in_m= s=3D600000; dynamic_snitch_update_interval_in_ms=3D100; endpoint_snitch=3DS= impleSnitch; hinted_handoff_enabled=3Dtrue; hinted_handoff_throttle_in_kb= =3D1024; incremental_backups=3Dfalse; index_summary_capacity_in_mb=3Dnull; = index_summary_resize_interval_in_minutes=3D60; inter_dc_tcp_nodelay=3Dfalse= ; internode_compression=3Dall; key_cache_save_period=3D14400; key_cache_siz= e_in_mb=3Dnull; listen_address=3Dlocalhost; max_hint_window_in_ms=3D1080000= 0; max_hints_delivery_threads=3D2; memtable_allocation_type=3Dheap_buffers;= native_transport_port=3D9042; num_tokens=3D256; partitioner=3Dorg.apache.c= assandra.dht.Murmur3Partitioner; permissions_validity_in_ms=3D2000; range_r= equest_timeout_in_ms=3D40000; read_request_timeout_in_ms=3D30000; request_s= cheduler=3Dorg.apache.cassandra.scheduler.NoScheduler; request_timeout_in_m= s=3D50000; row_cache_save_period=3D0; row_cache_size_in_mb=3D0; rpc_address= =3Dlocalhost; rpc_keepalive=3Dtrue; rpc_port=3D9160; rpc_server_type=3Dsync= ; saved_caches_directory=3Dz:/cassandra/saved_caches; seed_provider=3D[{cla= ss_name=3Dorg.apache.cassandra.locator.SimpleSeedProvider, parameters=3D[{s= eeds=3D127.0.0.1}]}]; server_encryption_options=3D; snapshot_befo= re_compaction=3Dfalse; ssl_storage_port=3D7001; sstable_preemptive_open_int= erval_in_mb=3D50; start_native_transport=3Dtrue; start_rpc=3Dtrue; storage_= port=3D7000; thrift_framed_transport_size_in_mb=3D15; tombstone_failure_thr= eshold=3D100000; tombstone_warn_threshold=3D1000; trickle_fsync=3Dfalse; tr= ickle_fsync_interval_in_kb=3D10240; truncate_request_timeout_in_ms=3D90000;= write_request_timeout_in_ms=3D30000] > INFO [main] 2015-07-18 21:30:05,683 MessagingService.java:480 - Starting= Messaging Service on port 7000 > ERROR [NonPeriodicTasks:1] 2015-07-18 21:30:05,685 SSTableDeletingTask.ja= va:85 - Unable to delete z:\cassandra\data\system\compaction_history-b4dbb7= b4dc493fb5b3bfce6e434832ca\system-compaction_history-tmplink-ka-1539-Data.d= b (it will be removed on server restart; we'll also retry after GC) > ERROR [main] 2015-07-18 21:30:05,747 CassandraDaemon.java:541 - Exception= encountered during startup > java.lang.RuntimeException: java.io.FileNotFoundException: z:\cassandra\d= ata\system\local-7ad54392bcdd35a684174e047860b377\system-local-tmplink-ka-3= 38-Data.db (The system cannot find the file specified) > =09at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(= CompressedRandomAccessReader.java:52) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createP= ooledReader(CompressedPoolingSegmentedFile.java:95) ~[apache-cassandra-2.1.= 8.jar:2.1.8] > =09at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(Poolin= gSegmentedFile.java:62) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTa= bleReader.java:1822) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.db.columniterator.SimpleSliceReader.(Sim= pleSliceReader.java:57) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createR= eader(SSTableSliceIterator.java:65) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.db.columniterator.SSTableSliceIterator.(= SSTableSliceIterator.java:42) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIte= rator(SliceQueryFilter.java:246) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator= (QueryFilter.java:62) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.db.CollationController.collectAllData(Collatio= nController.java:270) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.db.CollationController.getTopLevelColumns(Coll= ationController.java:65) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(Column= FamilyStore.java:1967) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFam= ilyStore.java:1810) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:357) ~[apache= -cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCo= mmand.java:85) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.cql3.statements.SelectStatement.readLocally(Se= lectStatement.java:309) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.cql3.statements.SelectStatement.executeInterna= l(SelectStatement.java:328) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.cql3.statements.SelectStatement.executeInterna= l(SelectStatement.java:67) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProce= ssor.java:317) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.db.SystemKeyspace.getBootstrapState(SystemKeys= pace.java:680) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.db.SystemKeyspace.bootstrapInProgress(SystemKe= yspace.java:695) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.service.StorageService.joinTokenRing(StorageSe= rvice.java:823) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.service.StorageService.initServer(StorageServi= ce.java:723) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.service.StorageService.initServer(StorageServi= ce.java:605) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.= java:378) [apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaem= on.java:524) [apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.j= ava:613) [apache-cassandra-2.1.8.jar:2.1.8] > Caused by: java.io.FileNotFoundException: z:\cassandra\data\system\local-= 7ad54392bcdd35a684174e047860b377\system-local-tmplink-ka-338-Data.db (The s= ystem cannot find the file specified) > =09at java.io.RandomAccessFile.open(Native Method) ~[na:1.8.0_11] > =09at java.io.RandomAccessFile.(RandomAccessFile.java:236) ~[na:1.8= .0_11] > =09at org.apache.cassandra.io.util.RandomAccessReader.(RandomAccess= Reader.java:64) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.io.compress.CompressedRandomAccessReader.(CompressedRandomAccessReader.java:70) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(= CompressedRandomAccessReader.java:48) ~[apache-cassandra-2.1.8.jar:2.1.8] > =09... 26 common frames omitted > INFO [CompactionExecutor:1] 2015-07-18 21:30:05,819 CompactionTask.java:= 274 - Compacted 4 sstables to [z:\cassandra\data\system\compaction_history-= b4dbb7b4dc493fb5b3bfce6e434832ca\system-compaction_history-ka-1539,]. 13,3= 59 bytes to 12,686 (~94% of original) in 565ms =3D 0.021413MB/s. 134 total= partitions merged to 134. Partition merge counts were {1:134, } > INFO [StorageServiceShutdownHook] 2015-07-18 21:30:05,833 Gossiper.java:= 1410 - Announcing shutdown > INFO [CompactionExecutor:2] 2015-07-18 21:30:06,023 CompactionTask.java:= 274 - Compacted 4 sstables to [z:\cassandra\data\system\local-7ad54392bcdd3= 5a684174e047860b377\system-local-ka-338,]. 11,594 bytes to 5,856 (~50% of = original) in 433ms =3D 0.012898MB/s. 4 total partitions merged to 1. Part= ition merge counts were {4:1, } > ERROR [NonPeriodicTasks:1] 2015-07-18 21:30:06,100 SSTableDeletingTask.ja= va:85 - Unable to delete z:\cassandra\data\system\local-7ad54392bcdd35a6841= 74e047860b377\system-local-tmplink-ka-338-Data.db (it will be removed on se= rver restart; we'll also retry after GC) > ERROR [NonPeriodicTasks:1] 2015-07-18 21:30:06,560 SSTableDeletingTask.ja= va:85 - Unable to delete z:\cassandra\data\system\compaction_history-b4dbb7= b4dc493fb5b3bfce6e434832ca\system-compaction_history-tmplink-ka-1539-Data.d= b (it will be removed on server restart; we'll also retry after GC) > ERROR [NonPeriodicTasks:1] 2015-07-18 21:30:06,564 SSTableDeletingTask.ja= va:85 - Unable to delete z:\cassandra\data\system\local-7ad54392bcdd35a6841= 74e047860b377\system-local-tmplink-ka-338-Data.db (it will be removed on se= rver restart; we'll also retry after GC) > INFO [StorageServiceShutdownHook] 2015-07-18 21:30:07,866 MessagingServi= ce.java:708 - Waiting for messaging service to quiesce > INFO [ACCEPT-localhost/127.0.0.1] 2015-07-18 21:30:07,867 MessagingServi= ce.java:958 - MessagingService has terminated the accept() thread > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)