cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Surbhi Gupta <surbhi.gupt...@gmail.com>
Subject Re: Bootstraping is failing
Date Thu, 07 May 2020 21:22:04 GMT
So after failed bootstrapped , if we start cassandra again on the new node
, will it resume bootstrap or will it start over?

On Thu, 7 May 2020 at 13:32, Adam Scott <adam.c.scott@gmail.com> wrote:

> I recommend it on all nodes.  This will eliminate that as a source of
> trouble further on down the road.
>
>
> On Thu, May 7, 2020 at 1:30 PM Surbhi Gupta <surbhi.gupta01@gmail.com>
> wrote:
>
>> streaming_socket_timeout_in_ms is 24 hour.
>>   So tcp settings should be changed on the new bootstrap node or on all
>> nodes ?
>>
>>
>> On Thu, 7 May 2020 at 13:23, Adam Scott <adam.c.scott@gmail.com> wrote:
>>
>>>
>>> *edit
>>> /etc/sysctl.confnet.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_probes=3net.ipv4.tcp_keepalive_intvl=10*
>>> then run sysctl -p to cause the kernel to reload the settings
>>>
>>> 5 minutes (300) seconds is probably too long.
>>>
>>> On Thu, May 7, 2020 at 1:09 PM Surbhi Gupta <surbhi.gupta01@gmail.com>
>>> wrote:
>>>
>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_time
>>>>
>>>> 300
>>>>
>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_intvl
>>>>
>>>> 30
>>>>
>>>> [root@abc cassandra]# cat /proc/sys/net/ipv4/tcp_keepalive_probes
>>>>
>>>> 9
>>>>
>>>> On Thu, 7 May 2020 at 12:32, Adam Scott <adam.c.scott@gmail.com> wrote:
>>>>
>>>>> Maybe a firewall killing a connection?
>>>>>
>>>>> What does the following show?
>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_time
>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_intvl
>>>>> cat /proc/sys/net/ipv4/tcp_keepalive_probes
>>>>>
>>>>> On Thu, May 7, 2020 at 10:31 AM Surbhi Gupta <surbhi.gupta01@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We are trying to expand a datacenter and trying to add nodes but
when
>>>>>> node is bootstrapping , it goes half way through and then fail with
below
>>>>>> error, We have increased stremthroughput from 200 to 400 when we
were
>>>>>> trying for the 2nd time but still it failed. We are on 3.11.0 , using
G1GC
>>>>>> with 31GB heap.
>>>>>>
>>>>>> ERROR [MessagingService-Incoming-/10.X.X.X] 2020-05-07 09:42:38,933
>>>>>> CassandraDaemon.java:228 - Exception in thread
>>>>>> Thread[MessagingService-Incoming-/10.X.X.X,main]
>>>>>>
>>>>>> java.io.IOError: java.io.EOFException: Stream ended prematurely
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:227)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:215)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:839)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:814)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:425)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:434)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:371)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.net.MessageIn.read(MessageIn.java:123)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:192)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:180)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>> Caused by: java.io.EOFException: Stream ended prematurely
>>>>>>
>>>>>>         at
>>>>>> net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:218)
>>>>>> ~[lz4-1.3.0.jar:na]
>>>>>>
>>>>>>         at
>>>>>> net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:150)
>>>>>> ~[lz4-1.3.0.jar:na]
>>>>>>
>>>>>>         at
>>>>>> net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:117)
>>>>>> ~[lz4-1.3.0.jar:na]
>>>>>>
>>>>>>         at
>>>>>> java.io.DataInputStream.readFully(DataInputStream.java:195) ~[na:1.8.0_242]
>>>>>>
>>>>>>         at
>>>>>> java.io.DataInputStream.readFully(DataInputStream.java:169) ~[na:1.8.0_242]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:437)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:245)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.readComplexColumn(UnfilteredSerializer.java:665)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$deserializeRowBody$1(UnfilteredSerializer.java:606)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1242)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1197)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at org.apache.cassandra.db.Columns.apply(Columns.java:377)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:600)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:475)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:431)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:222)
>>>>>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>>>>>>
>>>>>>         ... 11 common frames omitted
>>>>>>
>>>>>> Thanks
>>>>>> Surbhi
>>>>>>
>>>>>

Mime
View raw message