cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Terry Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10967) bootstrap fail with “Streaming error occurred”
Date Tue, 05 Jan 2016 07:03:39 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082542#comment-15082542
] 

Terry Ma commented on CASSANDRA-10967:
--------------------------------------

nodetool status feeds
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                     
         Rack
UN  172.31.20.223  665.85 GB  256          28.8%             e6c800d7-15ab-4b4b-8c0f-43a10df02fee
 rack4
UN  172.31.15.14   527.74 GB  256          28.8%             06a65c50-7a37-40df-9e05-7c84bc751e20
 rack5
UN  172.31.8.188   436.05 GB  256          26.4%             b7dbfb00-6fe0-41b7-ae48-b38ba9fa95f9
 rack1
UN  172.31.8.187   637.19 GB  256          28.5%             1666ea2e-fef1-42ef-9a66-de4d3c6e2704
 rack2
UN  172.31.16.232  497.47 GB  256          29.2%             ad8ece0d-c2d9-4618-8c2f-e13e12ac110b
 rack6
UN  172.31.4.135   496.75 GB  256          28.7%             3c74e65f-93fd-4405-99b1-a5b9453d554c
 rack7
UJ  172.31.4.134   39.98 GB   256          ?                 015f2a9a-a8d7-4c34-8936-184f35b56af6
 rack8
UN  172.31.20.224  524.46 GB  256          29.6%             44743768-fc48-45ac-a0cd-6c9027e33afc
 rack3

nodetool netstats
Mode: JOINING
Bootstrap 366d1720-b376-11e5-9aac-bbecd825311e
    /172.31.8.188
        Receiving 50 files, 48498301504 bytes total. Already received 0 files, 0 bytes total
    /172.31.20.223
        Receiving 56 files, 80283672656 bytes total. Already received 0 files, 0 bytes total
    /172.31.20.224
        Receiving 49 files, 88426202890 bytes total. Already received 0 files, 0 bytes total
    /172.31.16.232
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name                    Active   Pending      Completed
Large messages                  n/a         0              0
Small messages                  n/a         0      138949147
Gossip messages                 n/a         0         296937

> bootstrap fail with “Streaming error occurred”
> ----------------------------------------------
>
>                 Key: CASSANDRA-10967
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10967
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Streaming and Messaging
>         Environment: os: ubuntu 14.04.2 LTS (ec2)
>            Reporter: Terry Ma
>         Attachments: debug.log, system.log
>
>
> I installed  cassandra 2.2.4 from datastax apt source
> I have a cassandra cluster in aws ec2. the cluster has 7 nodes now. and I want to add
a new node to this cluster. but there is error occured with bootstrap. the bootstrap can not
finish sucessfully.
> I got the following errors on the new node:
> {code}
> WARN  [STREAM-IN-/172.31.4.135] 2016-01-01 15:23:44,228 StreamSession.java:644 - [Stream
#b1cc3600-b054-11e5-80c
> 4-21d1c7c11a01] Retrying for following error
> java.lang.ArrayIndexOutOfBoundsException: null
> ERROR [Thread-571] 2016-01-01 15:23:44,228 CassandraDaemon.java:185 - Exception in thread
Thread[Thread-571,5,ma
> in]
> java.lang.RuntimeException: java.lang.InterruptedException
>         at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.2.4.jar:
> 2.2.4]
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]
> Caused by: java.lang.InterruptedException: null
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer
> .java:1220) ~[na:1.8.0_66]
>         at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
~[na:1.8.0_66]
>         at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:350) ~[na:1.8.0_66]
>         at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStrea
> m.java:176) ~[apache-cassandra-2.2.4.jar:2.2.4]
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.4.jar:
> 2.2.4]
> {code}
> and I also got following errors: 
> {code}
> WARN  [STREAM-IN-/172.31.20.223] 2016-01-01 15:37:30,941 StreamSession.java:644 - [Stream
#b1cc3600-b054-11e5-80
> c4-21d1c7c11a01] Retrying for following error
> java.lang.RuntimeException: Last written key DecoratedKey(-8466153190082758358, 000000000000356a)
>= current key
>  DecoratedKey(-9223372036854775808, ) writing into /data_lvm/cassandra/data/feeds/inbox-eb873af0a19711e5ade0432b
> 31304f95/tmp-la-309-big-Data.db
> {code}
> I tried many things to do bootstrap, like cleanuped all nodes, scrubed all nodes, repaired
all nodes. but It still fails with these errors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message