Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 8FB761639EC for ; Tue, 25 Jul 2017 02:52:04 +0200 (CEST) Received: (qmail 88909 invoked by uid 500); 25 Jul 2017 00:52:03 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 88898 invoked by uid 99); 25 Jul 2017 00:52:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Jul 2017 00:52:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 0BC5D1800EA for ; Tue, 25 Jul 2017 00:52:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id lpk4mmRPtDOr for ; Tue, 25 Jul 2017 00:52:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id EFB705FCC6 for ; Tue, 25 Jul 2017 00:52:00 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 7690AE0630 for ; Tue, 25 Jul 2017 00:52:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 2F19923EFC for ; Tue, 25 Jul 2017 00:52:00 +0000 (UTC) Date: Tue, 25 Jul 2017 00:52:00 +0000 (UTC) From: "Kurt Greaves (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-13726) bootstrap streaming failed suspect due to secondary index MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099337#comment-16099337 ] Kurt Greaves commented on CASSANDRA-13726: ------------------------------------------ You need to check both sides of the stream for errors. Have a look at the corresponding log on "xxx.xxx.xxx.xxx" for errors that could explain the failure. > bootstrap streaming failed suspect due to secondary index > --------------------------------------------------------- > > Key: CASSANDRA-13726 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13726 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.1.18 > Reporter: peng xiao > > Hi, > We have a Cassandra DC with 31 nodes,but it's always failed when adding the 32nd node. > The following is the error log.And we suspect it's due to the secondary index,but it never happens before.Could you please advise? > Thanks > INFO [CompactionExecutor:3] 2017-07-24 22:07:40,774 ColumnFamilyStore.java:917 - Enqueuing flush of compactions_in_progress: 149 (0%) on-heap, 20 (0%) off-heap > INFO [MemtableFlushWriter:286] 2017-07-24 22:07:40,775 Memtable.java:347 - Writing Memtable-compactions_in_progress@581635159(0.008KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit) > INFO [MemtableFlushWriter:286] 2017-07-24 22:07:40,775 Memtable.java:382 - Completed flushing /data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-1159-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1500866839309, position=11929032) > INFO [CompactionExecutor:3] 2017-07-24 22:07:40,796 CompactionTask.java:274 - Compacted 15 sstables to [/data/cassandra/data/ecommercedata/ecommerce_baitiao_bill-c1a7f8d0660511e781e2077cc592d79d/ecommercedata-ecommerce_baitiao_bill-ka-40,]. 293,793 bytes to 288,251 (~98% of original) in 108ms = 2.545348MB/s. 1,024 total partitions merged to 1,016. Partition merge counts were {1:1008, 2:8, } > INFO [StreamReceiveTask:5] 2017-07-24 22:07:43,351 ColumnFamilyStore.java:917 - Enqueuing flush of ecommerce_baitiao_record.baitiao_state: 2228978 (0%) on-heap, 6183142 (0%) off-heap > INFO [CompactionExecutor:3] 2017-07-24 22:07:43,352 ColumnFamilyStore.java:917 - Enqueuing flush of compactions_in_progress: 349 (0%) on-heap, 266 (0%) off-heap > INFO [MemtableFlushWriter:287] 2017-07-24 22:07:43,353 Memtable.java:347 - Writing Memtable-ecommerce_baitiao_record.baitiao_state@1651345625(4.792MiB serialized bytes, 89948 ops, 0%/0% of on/off-heap limit) > INFO [MemtableFlushWriter:286] 2017-07-24 22:07:43,353 Memtable.java:347 - Writing Memtable-compactions_in_progress@284311350(0.197KiB serialized bytes, 9 ops, 0%/0% of on/off-heap limit) > INFO [MemtableFlushWriter:286] 2017-07-24 22:07:43,353 Memtable.java:382 - Completed flushing /data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-1160-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1500866839309, position=13336501) > INFO [MemtableFlushWriter:287] 2017-07-24 22:07:43,542 Memtable.java:382 - Completed flushing /data/cassandra/data/ecommercedata/ecommerce_baitiao_record-c94692e012cd11e682501f0d5e9f1e54/ecommercedata-ecommerce_baitiao_record.baitiao_state-tmp-ka-15-Data.db (1.593MiB) for commitlog position ReplayPosition(segmentId=1500866839309, position=13336501) > INFO [StreamReceiveTask:5] 2017-07-24 22:07:43,558 SecondaryIndexManager.java:174 - Index build of [ecommerce_baitiao_record.baitiao_state] complete > INFO [StreamReceiveTask:5] 2017-07-24 22:07:43,558 StreamResultFuture.java:181 - [Stream #15d5ffa0-7020-11e7-8b09-d510dd9f6293] Session with /xxx.xxx.xxx.xxx is complete > WARN [StreamReceiveTask:5] 2017-07-24 22:07:43,559 StreamResultFuture.java:208 - [Stream #15d5ffa0-7020-11e7-8b09-d510dd9f6293] Stream failed > ERROR [main] 2017-07-24 22:07:43,561 CassandraDaemon.java:583 - Exception encountered during startup > java.lang.RuntimeException: Error during boostrap: Stream failed > at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:87) ~[apache-cassandra-2.1.18.jar:2.1.18] > at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1166) ~[apache-cassandra-2.1.18.jar:2.1.18] > at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:944) ~[apache-cassandra-2.1.18.jar:2.1.18] > at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740) ~[apache-cassandra-2.1.18.jar:2.1.18] > at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617) ~[apache-cassandra-2.1.18.jar:2.1.18] > at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391) [apache-cassandra-2.1.18.jar:2.1.18] > at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) [apache-cassandra-2.1.18.jar:2.1.18] > at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) [apache-cassandra-2.1.18.jar:2.1.18] > Caused by: org.apache.cassandra.streaming.StreamException: Stream failed > at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-2.1.18.jar:2.1.18] > at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na] > at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] > at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-16.0.jar:na] > at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.jar:na] > at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-16.0.jar:na] > at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:209) ~[apache-cassandra-2.1.18.jar:2.1.18] > at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185) ~[apache-cassandra-2.1.18.jar:2.1.18] > at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:413) ~[apache-cassandra-2.1.18.jar:2.1.18] > at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:700) ~[apache-cassandra-2.1.18.jar:2.1.18] > at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:661) ~[apache-cassandra-2.1.18.jar:2.1.18] > at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.18.jar:2.1.18] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_131] > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131] > WARN [StorageServiceShutdownHook] 2017-07-24 22:07:43,564 Gossiper.java:1462 - No local state or state is in silent shutdown, not announcing shutdown > INFO [StorageServiceShutdownHook] 2017-07-24 22:07:43,564 MessagingService.java:734 - Waiting for messaging service to quiesce > INFO [ACCEPT-/172.16.135.52] 2017-07-24 22:07:43,565 MessagingService.java:1020 - MessagingService has terminated the accept() thread > INFO [CompactionExecutor:3] 2017-07-24 22:07:43,571 CompactionTask.java:141 - Compacting [SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-10-Data.db'), SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-12-Data.db'), SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-11-Data.db'), SSTableReader(path='/data/cassandra/data/peoplephonenumber/people_by_peopleid-38befca077fd11e5aa40f56762d1c5e6/peoplephonenumber-people_by_peopleid-ka-14-Data.db')] -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org For additional commands, e-mail: commits-help@cassandra.apache.org