cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-10130) Node failure during 2i update after streaming can have incomplete 2i when restarted
Date Fri, 02 Jun 2017 12:38:04 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034602#comment-16034602
] 

Paulo Motta edited comment on CASSANDRA-10130 at 6/2/17 12:38 PM:
------------------------------------------------------------------

bq. I think in such case the new Index initialization task would take care of indexing, so
it doesn't really matter if the new index misses the sstable notification. So, to cover against
the specific race you mentioned, we can just filter out in SIM#handleNotification(), when
receiving the SSTableAddedNotification, all indexes not marked as building, as we can assume
those missed the first notification because not yet registered (and being just registered,
the initialization task will eventually take care of any initial indexing).

It's still possible that an index is created after the {{SSTableAddedNotification}} but before
all sstables are added to the tracker, in this case the initial index rebuild will not index
the sstables that were added and filtering all indexes marked as building will not re-mark
the new index as building since it may be still doing the initial rebuild:
1. {{SSTableBeforeAddNotification}} is received by SIM
3. new index is created (it will not index the new sstables since they are not in the tracker
yet) and mark as building
4. New sstables are added to the tracker
5. {{SSTableAddeNotification}} is received and index rebuild of new sstables is triggered
- the index is not re-marked as building because it's already marked as building by the index
creation
6. original index rebuild is finished and index is marked as built
7. index rebuild of new sstables fail but index is marked as built

This case is extremely unlikely to happen but shows the fragility of this approach, which
may be prone to other races we're not aware of.

bq.I think the race above can be solved easily without adding columns to the system table.
Other than that, let's not forget the pendingBuilds counter was needed to protect us not just
against concurrent building of multiple indexes, but also against concurrent building of multiple
sstables "batches" for the same index.

My suggestion was not to add new columns to the system table, but rather overload the existing
columns with the following semantics:
- When an index is created for the first time for a table add new entry {{INSERT INTO system.IndexInfo
(table_name, index_name) VALUES ("table", "table")}}
- When new sstables are received remove this entry from the index while these sstables are
being indexed
- On restart, if this entry is not present it means index rebuild failure for all indexes
which need rebuild

To protect against concurrent building of multiple sstables "batches" for the same index,
we could still have a pendingBuilds counter but a single counter for all indexes rather than
one counter per index. For manual rebuilds we could simply prevent a new index rebuild if
there's one already running, since it doesn't make much sense to do simultaneous full rebuilds
of the same indexes.

In any case this is just a suggestion, I'm fine if we go with other approach (such as saving
which indices were marked on the first notification) or decide this race is too unlikely to
bother doing anything.


was (Author: pauloricardomg):
bq. I think in such case the new Index initialization task would take care of indexing, so
it doesn't really matter if the new index misses the sstable notification. So, to cover against
the specific race you mentioned, we can just filter out in SIM#handleNotification(), when
receiving the SSTableAddedNotification, all indexes not marked as building, as we can assume
those missed the first notification because not yet registered (and being just registered,
the initialization task will eventually take care of any initial indexing).

It's still possible that an index is created after the {{SSTableAddedNotification}} but before
all sstables are added to the tracker, in this case the initial index rebuild will not index
the sstables that were added and filtering all indexes marked as building will not re-mark
the new index as building since it may be still doing the initial rebuild:
1. {{SSTableBeforeAddNotification}} is received by SIM
3. new index is created (it will not index the new sstables since they are not in the tracker
yet) and mark as building
4. New sstables are added to the tracker
5. {{SSTableAddeNotification}} is received and index rebuild of new sstables is triggered
- the index is not re-marked as building because it's already marked as building by the index
creation
6. original index rebuild is finished and index is marked as built
7. index rebuild of new sstables fail but index is marked as built

This case is extremely unlikely to happen but shows the fragility of this approach, which
may be prone to other races we're not aware of.

bq.I think the race above can be solved easily without adding columns to the system table.
Other than that, let's not forget the pendingBuilds counter was needed to protect us not just
against concurrent building of multiple indexes, but also against concurrent building of multiple
sstables "batches" for the same index.

My suggestion was not to add new columns to the system table, but rather overload the existing
columns with the following semantics:
- When an index is created add new entry {{INSERT INTO system.IndexInfo (table_name, index_name)
VALUES ("table", "table")}}
- When new sstables are received remove this entry from the index while these sstables are
being indexed
- On restart, if this entry is not present it means index rebuild failure for all indexes
which need rebuild

To protect against concurrent building of multiple sstables "batches" for the same index,
we could still have a pendingBuilds counter but a single counter for all indexes rather than
one counter per index. For manual rebuilds we could simply prevent a new index rebuild if
there's one already running, since it doesn't make much sense to do simultaneous full rebuilds
of the same indexes.

In any case this is just a suggestion, I'm fine if we go with other approach (such as saving
which indices were marked on the first notification) or decide this race is too unlikely to
bother doing anything.

> Node failure during 2i update after streaming can have incomplete 2i when restarted
> -----------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-10130
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10130
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Coordination
>            Reporter: Yuki Morishita
>            Assignee: Andrés de la Peña
>            Priority: Minor
>
> Since MV/2i update happens after SSTables are received, node failure during MV/2i update
can leave received SSTables live when restarted while MV/2i are partially up to date.
> We can add some kind of tracking mechanism to automatically rebuild at the startup, or
at least warn user when the node restarts.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message