cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Byrd (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-13569) Schedule schema pulls just once per endpoint
Date Fri, 02 Jun 2017 20:47:04 GMT


Matt Byrd commented on CASSANDRA-13569:

Sure n.p []
Yes, so adding jitter in MIGRATION_DELAY_IN_MS could help when we're past: 
| runtimeMXBean.getUptime() < MIGRATION_DELAY_IN_MS)
However it doesn't help on startup.
Initially in trying to solve CASSANDRA-11748, I did also think about adding random the delay
for even this branch (where we've only been up a short amount of time).
This just didn't seem that straightforward to do and also guarantee that we wouldn't hit the
problem described in CASSANDRA-11748.
How do you know what is enough random delay? what if you actually delay getting the schema

I suppose the concerns in this ticket are similar but not exactly the same as CASSANDRA-11748,
though I admit that rate limiting the number of schema pulls per endpoint to one at a time
seems sensible and might possibly help a bit with CASSANDRA-11748.
The schema is being pulled repeatedly from the same instances in CASSANDRA-11748, but I'm
not sure rate limiting alone as described above will definitely solve it, perhaps it will
make it less likely to OOM, but we're still going to have a lot of incoming serialised schemas
from lots of nodes and we're still left with this sort of rough limit to scalability of "number
of nodes * size of serialised schema" (albeit perhaps with a different threshold).

Maybe some upcoming changes in CASSANDRA-10699 and related tickets may make the problem CASSANDRA-11748
even less likely, since part of the problem is that we're sending the entire serialised schema
inside a mutation, which can end up being quite large if you have lots of tables or lots of
columns in lots of tables.

Also, for reference I believe the migration delay was added in the following ticket, in order
to give a schema alteration sufficient time to propagate from the node where it changed, 
and not have a migration task race with this change and pull the whole schema instead of receive
the delta:

> Schedule schema pulls just once per endpoint
> --------------------------------------------
>                 Key: CASSANDRA-13569
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Distributed Metadata
>            Reporter: Stefan Podkowinski
>            Assignee: Stefan Podkowinski
>             Fix For: 3.0.x, 3.11.x, 4.x
> Schema mismatches detected through gossip will get resolved by calling {{MigrationManager.maybeScheduleSchemaPull}}.
This method may decide to schedule execution of {{MigrationTask}}, but only after using a
{{MIGRATION_DELAY_IN_MS = 60000}} delay (for reasons unclear to me). Meanwhile, as long as
the migration task hasn't been executed, we'll continue to have schema mismatches reported
by gossip and will have corresponding {{maybeScheduleSchemaPull}} calls, which will schedule
further tasks with the mentioned delay. Some local testing shows that dozens of tasks for
the same endpoint will eventually be executed and causing the same, stormy behavior for this
very endpoints.
> My proposal would be to simply not schedule new tasks for the same endpoint, in case
we still have pending tasks waiting for execution after {{MIGRATION_DELAY_IN_MS}}.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message