cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Kjellman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5129) newly bootstrapping nodes hang indefinitely in STATUS:BOOT while JOINING cluster
Date Thu, 14 Feb 2013 03:36:13 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578124#comment-13578124
] 

Michael Kjellman commented on CASSANDRA-5129:
---------------------------------------------

it appears this is related to secondary indexes. after the bootstrapping node finishes streaming
it submits an index build. This gets submitted but never makes any progress and hangs indefinitely.

{code}
 INFO [Thread-382] 2013-02-13 18:02:57,205 StreamInSession.java (line 199) Finished streaming
session 4ae0be23-75fb-11e2-ba65-8f73c0b9d93d from /10.138.12.10
 INFO [Thread-540] 2013-02-13 18:17:42,526 SecondaryIndexManager.java (line 137) Submitting
index build of [domain_metadata.classificationIdx, domain_metadata.domaintypeIdx] for data
in SSTableReader(path='/data/cassandra/brts/domain_metadata/brts-domain_metadata-ib-1-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-2-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-3-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-4-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-5-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-6-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-7-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-8-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-9-Data.db'),
SSTableReader(path='/data2/cassandra/brts/domain_metadata/brts-domain_metadata-ib-10-Data.db')
{code}
                
> newly bootstrapping nodes hang indefinitely in STATUS:BOOT while JOINING cluster  
> ----------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-5129
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5129
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.0
>         Environment: Ubuntu 12.04
>            Reporter: Michael Kjellman
>
> bootstrapping a new node causes it to hang indefinitely in STATUS:BOOT
> Nodes streaming to the new node report 
> {code}
> Mode: NORMAL
>  Nothing streaming to /10.8.30.16
> Not receiving any streams.
> Pool Name                    Active   Pending      Completed
> Commands                        n/a         0        1843990
> Responses                       n/a         2         661750
> {code}
> the node being streamed to stuck in the JOINING state reports:
> {code}
> Mode: JOINING
> Not sending any streams.
>  Nothing streaming from /10.8.30.103
>  Nothing streaming from /10.8.30.102
> Pool Name                    Active   Pending      Completed
> Commands                        n/a         0             10
> Responses                       n/a         0         613577
> {code}
> it appears that the nodes in the "nothing streaming" state never sends a "finished streaming"
to the joining node.
> no exceptions are thrown during the streaming on either node while the node is in this
state.
> {code:name="full gossip state of bootstrapping node"}
> /10.8.30.16
>   NET_VERSION:6
>   RELEASE_VERSION:1.2.0
>   STATUS:BOOT,127605887595351923798765477786913079289
>   RACK:RAC1
>   RPC_ADDRESS:0.0.0.0
>   DC:DC1
>   SCHEMA:5cd8420d-ce3c-3625-8293-67558a24816b
>   HOST_ID:e20817ce-7454-4dc4-a1c6-b1dec35c4491
>   LOAD:1.11824041581E11
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message