cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kurt Greaves (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13726) bootstrap streaming failed suspect due to secondary index
Date Tue, 25 Jul 2017 00:52:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099337#comment-16099337
] 

Kurt Greaves commented on CASSANDRA-13726:
------------------------------------------

You need to check both sides of the stream for errors. Have a look at the corresponding log
on "xxx.xxx.xxx.xxx" for errors that could explain the failure.

> bootstrap streaming failed suspect due to secondary index
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-13726
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13726
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Cassandra 2.1.18
>            Reporter: peng xiao
>
> Hi,
> We have a Cassandra DC with 31 nodes,but it's always failed when adding the 32nd node.
> The following is the error log.And we suspect it's due to the secondary index,but it
never happens before.Could you please advise?
> Thanks
> INFO  [CompactionExecutor:3] 2017-07-24 22:07:40,774 ColumnFamilyStore.java:917 - Enqueuing
flush of compactions_in_progress: 149 (0%) on-heap, 20 (0%) off-heap
> INFO  [MemtableFlushWriter:286] 2017-07-24 22:07:40,775 Memtable.java:347 - Writing Memtable-compactions_in_progress@581635159(0.008KiB
serialized bytes, 1 ops, 0%/0% of on/off-heap limit)
> INFO  [MemtableFlushWriter:286] 2017-07-24 22:07:40,775 Memtable.java:382 - Completed
flushing /data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-1159-Data.db
(0.000KiB) for commitlog position ReplayPosition(segmentId=1500866839309, position=11929032)
> INFO  [CompactionExecutor:3] 2017-07-24 22:07:40,796 CompactionTask.java:274 - Compacted
15 sstables to [/data/cassandra/data/ecommercedata/ecommerce_baitiao_bill-c1a7f8d0660511e781e2077cc592d79d/ecommercedata-ecommerce_baitiao_bill-ka-40,].
 293,793 bytes to 288,251 (~98% of original) in 108ms = 2.545348MB/s.  1,024 total partitions
merged to 1,016.  Partition merge counts were {1:1008, 2:8, }
> INFO  [StreamReceiveTask:5] 2017-07-24 22:07:43,351 ColumnFamilyStore.java:917 - Enqueuing
flush of ecommerce_baitiao_record.baitiao_state: 2228978 (0%) on-heap, 6183142 (0%) off-heap
> INFO  [CompactionExecutor:3] 2017-07-24 22:07:43,352 ColumnFamilyStore.java:917 - Enqueuing
flush of compactions_in_progress: 349 (0%) on-heap, 266 (0%) off-heap
> INFO  [MemtableFlushWriter:287] 2017-07-24 22:07:43,353 Memtable.java:347 - Writing Memtable-ecommerce_baitiao_record.baitiao_state@1651345625(4.792MiB
serialized bytes, 89948 ops, 0%/0% of on/off-heap limit)
> INFO  [MemtableFlushWriter:286] 2017-07-24 22:07:43,353 Memtable.java:347 - Writing Memtable-compactions_in_progress@284311350(0.197KiB
serialized bytes, 9 ops, 0%/0% of on/off-heap limit)
> INFO  [MemtableFlushWriter:286] 2017-07-24 22:07:43,353 Memtable.java:382 - Completed
flushing /data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-1160-Data.db
(0.000KiB) for commitlog position ReplayPosition(segmentId=1500866839309, position=13336501)
> INFO  [MemtableFlushWriter:287] 2017-07-24 22:07:43,542 Memtable.java:382 - Completed
flushing /data/cassandra/data/ecommercedata/ecommerce_baitiao_record-c94692e012cd11e682501f0d5e9f1e54/ecommercedata-ecommerce_baitiao_record.baitiao_state-tmp-ka-15-Data.db
(1.593MiB) for commitlog position ReplayPosition(segmentId=1500866839309, position=13336501)
> INFO  [StreamReceiveTask:5] 2017-07-24 22:07:43,558 SecondaryIndexManager.java:174 -
Index build of [ecommerce_baitiao_record.baitiao_state] complete
> INFO  [StreamReceiveTask:5] 2017-07-24 22:07:43,558 StreamResultFuture.java:181 - [Stream
#15d5ffa0-7020-11e7-8b09-d510dd9f6293] Session with /xxx.xxx.xxx.xxx is complete
> WARN  [StreamReceiveTask:5] 2017-07-24 22:07:43,559 StreamResultFuture.java:208 - [Stream
#15d5ffa0-7020-11e7-8b09-d510dd9f6293] Stream failed
> ERROR [main] 2017-07-24 22:07:43,561 CassandraDaemon.java:583 - Exception encountered
during startup
> java.lang.RuntimeException: Error during boostrap: Stream failed
>         at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:87) ~[apache-cassandra-2.1.18.jar:2.1.18]
>         at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1166)
~[apache-cassandra-2.1.18.jar:2.1.18]
>         at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:944)
~[apache-cassandra-2.1.18.jar:2.1.18]
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
~[apache-cassandra-2.1.18.jar:2.1.18]
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617)
~[apache-cassandra-2.1.18.jar:2.1.18]
>         at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391)
[apache-cassandra-2.1.18.jar:2.1.18]
>         at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566)
[apache-cassandra-2.1.18.jar:2.1.18]
>         at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655)
[apache-cassandra-2.1.18.jar:2.1.18]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>         at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-2.1.18.jar:2.1.18]
>         at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na]
>         at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
~[guava-16.0.jar:na]
>         at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-16.0.jar:na]
>         at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-16.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-16.0.jar:na]
>         at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:209)
~[apache-cassandra-2.1.18.jar:2.1.18]
>         at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185)
~[apache-cassandra-2.1.18.jar:2.1.18]
>         at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:413)
~[apache-cassandra-2.1.18.jar:2.1.18]
>         at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:700)
~[apache-cassandra-2.1.18.jar:2.1.18]
>         at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:661)
~[apache-cassandra-2.1.18.jar:2.1.18]
>         at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:179)
~[apache-cassandra-2.1.18.jar:2.1.18]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_131]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_131]
>         at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
> WARN  [StorageServiceShutdownHook] 2017-07-24 22:07:43,564 Gossiper.java:1462 - No local
state or state is in silent shutdown, not announcing shutdown
> INFO  [StorageServiceShutdownHook] 2017-07-24 22:07:43,564 MessagingService.java:734
- Waiting for messaging service to quiesce
> INFO  [ACCEPT-/172.16.135.52] 2017-07-24 22:07:43,565 MessagingService.java:1020 - MessagingService
has terminated the accept() thread
> INFO  [CompactionExecutor:3] 2017-07-24 22:07:43,571 CompactionTask.java:141 - Compacting
[SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-10-Data.db'),
SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-12-Data.db'),
SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-11-Data.db'),
SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-14-Data.db')]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message