incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ethan Rowe <et...@the-rowes.com>
Subject Re: New node unable to stream (0.8.5)
Date Thu, 15 Sep 2011 12:13:30 GMT
Here's a typical log slice (not terribly informative, I fear):

>  INFO [AntiEntropyStage:2] 2011-09-15 05:41:36,106 AntiEntropyService.java
> (l
> ine 884) Performing streaming repair of 1003 ranges with /10.34.90.8 for
> (299
> 90798416657667504332586989223299634,54296681768153272037430773234349600451]
>  INFO [AntiEntropyStage:2] 2011-09-15 05:41:36,427 StreamOut.java (line
> 181)
> Stream context metadata
> [/mnt/cassandra/data/events_production/FitsByShip-g-1
> 0-Data.db sections=88 progress=0/11707163 - 0%,
> /mnt/cassandra/data/events_pr
> oduction/FitsByShip-g-11-Data.db sections=169 progress=0/6133240 - 0%,
> /mnt/c
> assandra/data/events_production/FitsByShip-g-6-Data.db sections=1
> progress=0/
> 6918814 - 0%, /mnt/cassandra/data/events_production/FitsByShip-g-12-Data.db
> s
> ections=260 progress=0/9091780 - 0%], 4 sstables.
>  INFO [AntiEntropyStage:2] 2011-09-15 05:41:36,428 StreamOutSession.java
> (lin
> e 174) Streaming to /10.34.90.8
> ERROR [Thread-56] 2011-09-15 05:41:38,515 AbstractCassandraDaemon.java
> (line
> 139) Fatal exception in thread Thread[Thread-56,5,main]
> java.lang.NullPointerException
>         at
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpC
> onnection.java:174)
>         at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConn
> ection.java:114)



Not sure if the exception is related to the outbound streaming above; other
nodes are actively trying to stream to this node, so perhaps it comes from
those and temporal adjacency to the outbound stream is just coincidental.  I
have other snippets that look basically identical to the above, except if I
look at the logs to which this node is trying to stream, I see that it has
concurrently opened a stream in the other direction, which could be the one
that the exception pertains to.


On Thu, Sep 15, 2011 at 7:41 AM, Sylvain Lebresne <sylvain@datastax.com>wrote:

> On Thu, Sep 15, 2011 at 1:16 PM, Ethan Rowe <ethan@the-rowes.com> wrote:
> > Hi.
> >
> > We've been running a 7-node cluster with RF 3, QUORUM reads/writes in our
> > production environment for a few months.  It's been consistently stable
> > during this period, particularly once we got out maintenance strategy
> fully
> > worked out (per node, one repair a week, one major compaction a week, the
> > latter due to the nature of our data model and usage).  While this
> cluster
> > started, back in June or so, on the 0.7 series, it's been running 0.8.3
> for
> > a while now with no issues.  We upgraded to 0.8.5 two days ago, having
> > tested the upgrade in our staging cluster (with an otherwise identical
> > configuration) previously and verified that our application's various use
> > cases appeared successful.
> >
> > One of our nodes suffered a disk failure yesterday.  We attempted to
> replace
> > the dead node by placing a new node at OldNode.initial_token - 1 with
> > auto_bootstrap on.  A few things went awry from there:
> >
> > 1. We never saw the new node in bootstrap mode; it became available
> pretty
> > much immediately upon joining the ring, and never reported a "joining"
> > state.  I did verify that auto_bootstrap was on.
> >
> > 2. I mistakenly ran repair on the new node rather than removetoken on the
> > old node, due to a delightful mental error.  The repair got nowhere fast,
> as
> > it attempts to repair against the down node which throws an exception.
>  So I
> > interrupted the repair, restarted the node to clear any pending
> validation
> > compactions, and...
> >
> > 3. Ran removetoken for the old node.
> >
> > 4. We let this run for some time and saw eventually that all the nodes
> > appeared to be done various compactions and were stuck at streaming.
> Many
> > streams listed as open, none making any progress.
> >
> > 5.  I observed an Rpc-related exception on the new node (where the
> > removetoken was launched) and concluded that the streams were broken so
> the
> > process wouldn't ever finish.
> >
> > 6. Ran a "removetoken force" to get the dead node out of the mix.  No
> > problems.
> >
> > 7. Ran a repair on the new node.
> >
> > 8. Validations ran, streams opened up, and again things got stuck in
> > streaming, hanging for over an hour with no progress.
> >
> > 9. Musing that lingering tasks from the removetoken could be a factor, I
> > performed a rolling restart and attempted a repair again.
> >
> > 10. Same problem.  Did another rolling restart and attempted a fresh
> repair
> > on the most important column family alone.
> >
> > 11. Same problem.  Streams included CFs not specified, so I guess they
> must
> > be for hinted handoff.
> >
> > In concluding that streaming is stuck, I've observed:
> > - streams will be open to the new node from other nodes, but the new node
> > doesn't list them
> > - streams will be open to the other nodes from the new node, but the
> other
> > nodes don't list them
> > - the streams reported may make some initial progress, but then they hang
> at
> > a particular point and do not move on for an hour or more.
> > - The logs report repair-related activity, until NPEs on incoming TCP
> > connections show up, which appear likely to be the culprit.
>
> Can you send the stack trace from those NPE.
>
> >
> > I can provide more exact details when I'm done commuting.
> >
> > With streaming broken on this node, I'm unable to run repairs, which is
> > obviously problematic.  The application didn't suffer any operational
> issues
> > as a consequence of this, but I need to review the overnight results to
> > verify we're not suffering data loss (I doubt we are).
> >
> > At this point, I'm considering a couple options:
> > 1. Remove the new node and let the adjacent node take over its range
> > 2. Bring the new node down, add a new one in front of it, and properly
> > removetoken the problematic one.
> > 3. Bring the new node down, remove all its data except for the system
> > keyspace, then bring it back up and repair it.
> > 4. Revert to 0.8.3 and see if that helps.
> >
> > Recommendations?
> >
> > Thanks.
> > - Ethan
> >
>

Mime
View raw message