cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benedict (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-6937) On large insert, node is being marked down with Commit Log failure
Date Thu, 27 Mar 2014 00:32:15 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948706#comment-13948706
] 

Benedict commented on CASSANDRA-6937:
-------------------------------------

Silly question, but are they running a commit log archiving script, and if so, what does it
do?

> On large insert, node is being marked down with Commit Log failure
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-6937
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6937
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: 16 vcpu, 480GB SSD storage (used 7GB), 32GB RAM,  Cassandra 2.0.5.24,
hsha, datastax Java driver 2.0.1, vnodes
>            Reporter: jonathan lacefield
>            Assignee: Benedict
>            Priority: Minor
>         Attachments: cassandra.yaml copy, system.log
>
>
> during large insert, will get specs on insert load, nodes in the cluster are being marked
down.  creating defect for investigation.  
> error message in log file of one of the down nodes:
> ERROR [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:06,666 StorageService.java (line 364) Stopping
gossiper
> WARN [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:06,666 StorageService.java (line 278) Stopping
gossip by operator request
> INFO [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:06,666 Gossiper.java (line 1251) Announcing
shutdown
> INFO [FlushWriter:421] 2014-03-26 08:28:06,677 Memtable.java (line 371) Completed flushing
/var/lib/cassandra/data1/system/compactions_in_progress/system-compactions_in_progress-jb-198-Data.db
(166 bytes) for commitlog position ReplayPosition(segmentId=1395644152684, position=9577531)
> INFO [CompactionExecutor:1880] 2014-03-26 08:28:06,679 CompactionTask.java (line 115)
Compacting [SSTableReader(path='/var/lib/cassandra/data2/OpsCenter/rollups60/OpsCenter-rollups60-jb-18-Data.db'),
SSTableReader(path='/var/lib/cassandra/data1/OpsCenter/rollups60/OpsCenter-rollups60-jb-19-Data.db'),
SSTableReader(path='/var/lib/cassandra/data2/OpsCenter/rollups60/OpsCenter-rollups60-jb-17-Data.db'),
SSTableReader(path='/var/lib/cassandra/data1/OpsCenter/rollups60/OpsCenter-rollups60-jb-20-Data.db')]
> INFO [OptionalTasks:1] 2014-03-26 08:28:07,322 MeteredFlusher.java (line 63) flushing
high-traffic column family CFS(Keyspace='system', ColumnFamily='batchlog') (estimated 215899526
bytes)
> INFO [OptionalTasks:1] 2014-03-26 08:28:07,322 ColumnFamilyStore.java (line 784) Enqueuing
flush of Memtable-batchlog@74009414(39242953/215899526 serialized/live bytes, 239 ops)
> INFO [FlushWriter:422] 2014-03-26 08:28:07,323 Memtable.java (line 331) Writing Memtable-batchlog@74009414(39242953/215899526
serialized/live bytes, 239 ops)
> INFO [FlushWriter:422] 2014-03-26 08:28:07,370 Memtable.java (line 371) Completed flushing
/var/lib/cassandra/data1/system/batchlog/system-batchlog-jb-1-Data.db (204617 bytes) for commitlog
position ReplayPosition(segmentId=1395644152684, position=17705519)
> INFO [MemoryMeter:1] 2014-03-26 08:28:08,296 Memtable.java (line 449) CFS(Keyspace='system',
ColumnFamily='batchlog') liveRatio is 3.2515027780484544 (just-counted was 1.0013929201634204).
 calculation took 0ms for 12 cells
> ERROR [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,667 StorageService.java (line 369) Stopping
RPC server
> INFO [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,667 ThriftServer.java (line 141) Stop
listening to thrift clients
> ERROR [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,698 StorageService.java (line 374) Stopping
native transport
> INFO [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,765 Server.java (line 181) Stop listening
for CQL clients
> ERROR [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,766 CommitLog.java (line 379) Failed
to allocate new commit log segments. Commit disk failure policy is stop; terminating thread
> java.lang.AssertionError: attempted to delete non-existing file CommitLog-3-1395644152652.log



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message