phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (PHOENIX-4537) RegionServer initiating compaction can trigger schema migration and deadlock the system
Date Fri, 19 Jan 2018 20:55:00 GMT


Josh Elser commented on PHOENIX-4537:

bq. Are we saying that migration would happen before the first client connection? That's bad.
Can we change that?

There are two points here: one is migration (modification of the schema and contents of our
system tables). This happens on first client connection.

The issue here is that "renaming" of system tables from "SYSTEM.CATALOG" to the namespace-mapped
variants ("SYSTEM:CATALOG") is not presently subject to the same logic. It presently happens
for any Phoenix connection, client or server. I think we just haven't noticed this before.
The only time it would really cause an issue would be this deadlock where a compaction gets
queued which keeps the table from becoming disabled (until that RS is taken down)

> RegionServer initiating compaction can trigger schema migration and deadlock the system
> ---------------------------------------------------------------------------------------
>                 Key: PHOENIX-4537
>                 URL:
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Romil Choksi
>            Assignee: Josh Elser
>            Priority: Critical
>             Fix For: 5.0.0, 4.14.0
>         Attachments: PHOENIX-4537.001.patch
> [~sergey.soldatov] has been doing some great digging around a test failure we've been
seeing at $dayjob. The situation goes like this.
> 0. Run some arbitrary load
> 1. Stop HBase
> 2. Enable schema mapping ({{phoenix.schema.isNamespaceMappingEnabled=true}} and {{phoenix.schema.mapSystemTablesToNamespace=true}}
in hbase-site.xml)
> 3. Start HBase
> 4. Circumstantially, have the SYSTEM.CATALOG table need a compaction to run before a
client first connects
> When the RegionServer initiates the compaction, it will end up running {{UngroupedAggregateRegionObserver.clearTsOnDisabledIndexes}}
which opens a Phoenix connection. While the RegionServer won't upgrade system tables, it *will*
try to migrate them into the schema mapped variants (e.g. SYSTEM.CATALOG to SYSTEM:CATALOG).
> However, one of the first steps in the schema migration is to disable the SYSTEM.CATALOG
table. However, the SYSTEM.CATALOG table can't be disabled until the region is CLOSED, and
the region cannot be CLOSED until the compaction is finished. *deadlock*
> The "obvious" fix is to avoid RegionServers from triggering system table migrations,
but Sergey and [~elserj] both think that this will end badly (RegionServers falling over because
they expect the tables to be migrated and they aren't).
> Thoughts? [~ankit.singhal], [~jamestaylor], any others?

This message was sent by Atlassian JIRA

View raw message