phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <>
Subject [jira] [Updated] (PHOENIX-3811) Do not disable index on write failure by default
Date Wed, 10 May 2017 18:59:04 GMT


James Taylor updated PHOENIX-3811:
    Attachment: PHOENIX-3811_v2.patch

High level summary of changes: turns off automatic index rebuilding by default, leaves indexes
active upon a write failure, and provides a means of users replaying a commit after a write
failure to ensure the index is consistent with data table. Would you have any spare cycles
to review, [~tdsilva]? 

Here's some more detail on the changes:
- Turns off the background partial index rebuild/catchup task by default for a table. The
reason is that users will typically have some kind of retry strategy themselves (for example,
a message queue that retries). They need this as when a commit exception occurs, some of the
data rows may have been written while others will not have been (regardless of what state
the index is in wrt the data table). What ever retry mechanism is in use, these retries will
also get the index back in sync (see below for a new mechanism for mutable tables).
- Provides a means for the client to retry a commit at the timestamp at which it was originally
submitted. This is important for mutable data as otherwise the retried commits may overwrite
successful commits that occurred later. This is accomplished by a) including the server timestamp
at which the data rows were (or would have been) committed in CommitException and b) Adds
a new connection property, {{PhoenixRuntime.REPLAY_AT_ATTRIB}}, which specifies a timestamp
at which the commit will occur and tells the system to ignore later data updates (to ensure
your index remains in sync with your data table).
- Provides an option (the default) to keep an index active even after a write failure occurs.
Many use cases are essentially down without the secondary index in place and would rather
the index be behind by a few rows wrt the data table while the retries are occurring. This
option is configurable globally with the {{QueryServices.INDEX_FAILURE_DISABLE_INDEX }} config
property and on a table by table basis through the {{PhoenixIndexFailurePolicy.DISABLE_INDEX_ON_WRITE_FAILURE}}
table descriptor property.
- Provides an option to turn on the partial rebuild index task on a table-by-table basis (false
by default). This option is orthogonal now to whether an index remains active or is disabled.
Note that if the existing global {{PhoenixIndexFailurePolicy.INDEX_FAILURE_HANDLING_REBUILD_ATTRIB}}
config property is false, then the background thread won't run so the table property won't
matter. By default, the global property is true while the table-by-table property is false
to allow the user to turn the automatic rebuild on for a particular table.
- Lowers the default frequency at which we look for indexes which need to be partially rebuilt
from every 10 seconds to once per minute.
- Fixes MutableIndexFailureIT test failures and adds more for the above new options.

FYI, [~lhofhansl], [~apurtell], [~mvanwely].

> Do not disable index on write failure by default
> ------------------------------------------------
>                 Key: PHOENIX-3811
>                 URL:
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: James Taylor
>            Assignee: James Taylor
>             Fix For: 4.11.0
>         Attachments: PHOENIX-3811_v1.patch, PHOENIX-3811_v2.patch, PHOENIX-3811-wip1.patch,
PHOENIX-3811-wip2.patch, PHOENIX-3811-wip3.patch, PHOENIX-3811-wip4.patch, PHOENIX-3811-wip5.patch,
> We should provide a way to configure the system so that the server takes no specific
action when an index write fails. Since we always throw the write failure back to the client,
the client can often deal with failures more easily than the server since they have the batch
of mutations in memory. Often times, allowing access to an index that may be one batch behind
the data table is better than disabling it given the negative performance that will occur
while the index cannot be written to.

This message was sent by Atlassian JIRA

View raw message