cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Will Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-9846) CassandraDaemon fails to start due to FileNotFoundException
Date Mon, 20 Jul 2015 21:46:05 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634130#comment-14634130
] 

Will Zhang edited comment on CASSANDRA-9846 at 7/20/15 9:45 PM:
----------------------------------------------------------------

Hi Josh,
Yes I can confirm this is a mapped network drive. 

However, I have been running 2.1.3, 2.1.4 and 2.1.5 with this setup for the past year. The
server has managed 25mm+ row inserts and reads on top of that in this period. I've run many
drain/flush/repair operations also. Have to say, Cassandra is really great! This is my first
time seeing this error when trying to upgrade to 2.1.8. 

[CASSANDRA-8743|https://issues.apache.org/jira/browse/CASSANDRA-8743] seems to be a `DirectoryNotEmptyException`,
whereas I've seen a `FileNotFoundException`. Slight difference. Unfortunately I don't know
enough to say they are caused by the same issue I'm afraid.




was (Author: esvhd):
Hi Josh,
Yes I can confirm this is a mapped network drive. 

However, I have been running 2.1.3, 2.1.4 and 2.1.5 on with this setup for the past year.
The server has managed 25mm+ row inserts and reads on top of that in this period. In the past
year I've run many drain/flush/repair operations also. Have to say, Cassandra is really great!
This is my first time seeing this error when trying to upgrade to 2.1.8. 

[CASSANDRA-8743|https://issues.apache.org/jira/browse/CASSANDRA-8743] seems to be a `DirectoryNotEmptyException`,
whereas I've seen a `FileNotFoundException`. Slight difference. Unfortunately I don't know
enough to say they are caused by the same issue I'm afraid.



> CassandraDaemon fails to start due to FileNotFoundException
> -----------------------------------------------------------
>
>                 Key: CASSANDRA-9846
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9846
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Windows 7 64bit, 4-core, 24GB memory. Running single cluster.
> Java 1.8.0_45, python 2.7.10
>            Reporter: Will Zhang
>         Attachments: system.2.1.8.log
>
>
> Cassandra 2.1.8
> Server fails to start after seeing a `FileNotFoundException` in the logs.
> Before upgrading to `2.1.8`, I was running `2.1.5`. I ran the following before shutting
down `2.1.5`:
>     nodetool flush
>     nodetool drain
>     notetool repair
> When I tred to start 2.1.8 after a reboot, the following error is observed in the system.log
file. Server would not start after reach this point. I also attached the full error log which
shows two start attempts. 
> When I reverted back to 2.1.5, server started just fine, with no error in log.
> I saw the following issue reported but that was for 2.0.9. However, I have not seen this
error in 2.1.3, 2.1.4 or 2.1.5. 
> https://issues.apache.org/jira/browse/CASSANDRA-7780 
> Error log:
> {code:java}
> INFO  [main] 2015-07-18 21:30:05,582 YamlConfigurationLoader.java:135 - Node configuration:[authenticator=AllowAllAuthenticator;
authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
cas_contention_timeout_in_ms=5000; client_encryption_options=<REDACTED>; cluster_name=zhangwe;
column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=z:/cassandra/commitlog;
commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000;
compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32;
counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=40000;
cross_node_timeout=false; data_file_directories=[z:/cassandra/data]; disk_failure_policy=stop;
dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100;
endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60;
inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null;
listen_address=localhost; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers;
native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
permissions_validity_in_ms=2000; range_request_timeout_in_ms=40000; read_request_timeout_in_ms=30000;
request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=50000;
row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=localhost; rpc_keepalive=true;
rpc_port=9160; rpc_server_type=sync; saved_caches_directory=z:/cassandra/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider,
parameters=[{seeds=127.0.0.1}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false;
ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true;
start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000;
tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=90000;
write_request_timeout_in_ms=30000]
> INFO  [main] 2015-07-18 21:30:05,683 MessagingService.java:480 - Starting Messaging Service
on port 7000
> ERROR [NonPeriodicTasks:1] 2015-07-18 21:30:05,685 SSTableDeletingTask.java:85 - Unable
to delete z:\cassandra\data\system\compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca\system-compaction_history-tmplink-ka-1539-Data.db
(it will be removed on server restart; we'll also retry after GC)
> ERROR [main] 2015-07-18 21:30:05,747 CassandraDaemon.java:541 - Exception encountered
during startup
> java.lang.RuntimeException: java.io.FileNotFoundException: z:\cassandra\data\system\local-7ad54392bcdd35a684174e047860b377\system-local-tmplink-ka-338-Data.db
(The system cannot find the file specified)
> 	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:52)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createPooledReader(CompressedPoolingSegmentedFile.java:95)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:62)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1822)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:57)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:42)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:246)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:270)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1967)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1810)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:357) ~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:309)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:328)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:67)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:317)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.db.SystemKeyspace.getBootstrapState(SystemKeyspace.java:680)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.db.SystemKeyspace.bootstrapInProgress(SystemKeyspace.java:695)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:823)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:723) ~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:605) ~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) [apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:524) [apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:613) [apache-cassandra-2.1.8.jar:2.1.8]
> Caused by: java.io.FileNotFoundException: z:\cassandra\data\system\local-7ad54392bcdd35a684174e047860b377\system-local-tmplink-ka-338-Data.db
(The system cannot find the file specified)
> 	at java.io.RandomAccessFile.open(Native Method) ~[na:1.8.0_11]
> 	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:236) ~[na:1.8.0_11]
> 	at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:64)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:70)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:48)
~[apache-cassandra-2.1.8.jar:2.1.8]
> 	... 26 common frames omitted
> INFO  [CompactionExecutor:1] 2015-07-18 21:30:05,819 CompactionTask.java:274 - Compacted
4 sstables to [z:\cassandra\data\system\compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca\system-compaction_history-ka-1539,].
 13,359 bytes to 12,686 (~94% of original) in 565ms = 0.021413MB/s.  134 total partitions
merged to 134.  Partition merge counts were {1:134, }
> INFO  [StorageServiceShutdownHook] 2015-07-18 21:30:05,833 Gossiper.java:1410 - Announcing
shutdown
> INFO  [CompactionExecutor:2] 2015-07-18 21:30:06,023 CompactionTask.java:274 - Compacted
4 sstables to [z:\cassandra\data\system\local-7ad54392bcdd35a684174e047860b377\system-local-ka-338,].
 11,594 bytes to 5,856 (~50% of original) in 433ms = 0.012898MB/s.  4 total partitions merged
to 1.  Partition merge counts were {4:1, }
> ERROR [NonPeriodicTasks:1] 2015-07-18 21:30:06,100 SSTableDeletingTask.java:85 - Unable
to delete z:\cassandra\data\system\local-7ad54392bcdd35a684174e047860b377\system-local-tmplink-ka-338-Data.db
(it will be removed on server restart; we'll also retry after GC)
> ERROR [NonPeriodicTasks:1] 2015-07-18 21:30:06,560 SSTableDeletingTask.java:85 - Unable
to delete z:\cassandra\data\system\compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca\system-compaction_history-tmplink-ka-1539-Data.db
(it will be removed on server restart; we'll also retry after GC)
> ERROR [NonPeriodicTasks:1] 2015-07-18 21:30:06,564 SSTableDeletingTask.java:85 - Unable
to delete z:\cassandra\data\system\local-7ad54392bcdd35a684174e047860b377\system-local-tmplink-ka-338-Data.db
(it will be removed on server restart; we'll also retry after GC)
> INFO  [StorageServiceShutdownHook] 2015-07-18 21:30:07,866 MessagingService.java:708
- Waiting for messaging service to quiesce
> INFO  [ACCEPT-localhost/127.0.0.1] 2015-07-18 21:30:07,867 MessagingService.java:958
- MessagingService has terminated the accept() thread
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message