cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Zhou (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-13323) IncomingTcpConnection closed due to one bad message
Date Fri, 10 Mar 2017 23:53:04 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-13323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Simon Zhou updated CASSANDRA-13323:
-----------------------------------
    Description: 
We got this exception:
{code}
WARN  [MessagingService-Incoming-/****] 2017-02-14 17:33:33,177 IncomingTcpConnection.java:101
- UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId 2a3ab630-df74-11e6-9f81-b56251e1559e.
If a table was just created, this is likely due to the schema not being fully propagated.
 Please wait for schema agreement on table creation.
    at org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1336)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:660)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:635)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:131)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:113)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) ~[apache-cassandra-3.0.10.jar:3.0.10]
{code}

Also we saw this log in another host indicating it needs to re-connect:
{code}
INFO  [HANDSHAKE-/****] 2017-02-21 13:37:50,216 OutboundTcpConnection.java:515 - Handshaking
version with /****
{code}

The reason is that the node was receiving hinted data for a dropped table. This may happen
with other messages as well. On Cassandra side, IncomingTcpConnection shouldn't close on just
one bad message, even though it will be restarted soon later by SocketThread in MessagingService.

  was:
We got this exception:
{code}
WARN  [MessagingService-Incoming-/****] 2017-02-14 17:33:33,177 IncomingTcpConnection.java:101
- UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId 2a3ab630-df74-11e6-9f81-b56251e1559e.
If a table was just created, this is likely due to the schema not being fully propagated.
 Please wait for schema agreement on table creation.
    at org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1336)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:660)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:635)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:131)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:113)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) ~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
~[apache-cassandra-3.0.10.jar:3.0.10]
    at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) ~[apache-cassandra-3.0.10.jar:3.0.10]
{code}

Also we saw this log in another host indicating it needs to re-connect:
{code}
INFO  [HANDSHAKE-/****] 2017-02-21 13:37:50,216 OutboundTcpConnection.java:515 - Handshaking
version with /****
{code}

The reason is that another node was sending hinted data to this node. However the hinted data
was for a table that had been dropped. This may happen with other messages as well. On Cassandra
side, IncomingTcpConnection shouldn't close on just one bad message, even though it will be
restarted soon later by SocketThread in MessagingService.


> IncomingTcpConnection closed due to one bad message
> ---------------------------------------------------
>
>                 Key: CASSANDRA-13323
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13323
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Simon Zhou
>            Assignee: Simon Zhou
>             Fix For: 3.0.13
>
>         Attachments: CASSANDRA-13323-v1.patch
>
>
> We got this exception:
> {code}
> WARN  [MessagingService-Incoming-/****] 2017-02-14 17:33:33,177 IncomingTcpConnection.java:101
- UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId 2a3ab630-df74-11e6-9f81-b56251e1559e.
If a table was just created, this is likely due to the schema not being fully propagated.
 Please wait for schema agreement on table creation.
>     at org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1336)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:660)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:635)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:131)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:113)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) ~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
~[apache-cassandra-3.0.10.jar:3.0.10]
>     at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
~[apache-cassandra-3.0.10.jar:3.0.10]
> {code}
> Also we saw this log in another host indicating it needs to re-connect:
> {code}
> INFO  [HANDSHAKE-/****] 2017-02-21 13:37:50,216 OutboundTcpConnection.java:515 - Handshaking
version with /****
> {code}
> The reason is that the node was receiving hinted data for a dropped table. This may happen
with other messages as well. On Cassandra side, IncomingTcpConnection shouldn't close on just
one bad message, even though it will be restarted soon later by SocketThread in MessagingService.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message