phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajeshbabu Chintaguntla (JIRA)" <>
Subject [jira] [Commented] (PHOENIX-4027) Mark index as disabled during partial rebuild after configurable amount of time
Date Fri, 15 Sep 2017 11:10:00 GMT


Rajeshbabu Chintaguntla commented on PHOENIX-4027:

With this patch partial index rebuild make the index disabled forever in very easily in below
1) When we write the past data with row timestamp columns
2) Sometimes any region inconsistencies introduced take more than 30 mins

When the data is huge creating index or rebuilding complete index might take hours or days.
In such cases it's better to rebuild the index in intervals or batches than completely disabling.
[~samarthjain] [~jamestaylor] whyt?
                    if (EnvironmentEdgeManager.currentTimeMillis() - Math.abs(indexDisableTimestamp)
> indexDisableTimestampThreshold) {
                         * It has been too long since the index has been disabled and any
                         * attempts to reenable it likely will fail. So we are going to mark
                         * index as disabled and set the index disable timestamp to 0 so that
                         * rebuild task won't pick up this index again for rebuild.
                        try {
                            IndexUtil.updateIndexState(conn, indexTableFullName, PIndexState.DISABLE,
                            LOG.error("Unable to rebuild index " + indexTableFullName
                                    + ". Won't attempt again since index disable timestamp
is older than current time by "
                                    + indexDisableTimestampThreshold
                                    + " milliseconds. Manual intervention needed to re-build
the index");
                        } catch (Throwable ex) {
                                "Unable to mark index " + indexTableFullName + " as disabled.",
                        continue; // don't attempt another rebuild irrespective of whether
                                  // updateIndexState worked or not

> Mark index as disabled during partial rebuild after configurable amount of time
> -------------------------------------------------------------------------------
>                 Key: PHOENIX-4027
>                 URL:
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: James Taylor
>            Assignee: Samarth Jain
>             Fix For: 4.12.0, 4.11.1
>         Attachments: PHOENIX-4027_addendum.patch, PHOENIX-4027.patch
> Instead of marking an index as permanently disabled in the partial index rebuilder when
a failure occurs, we should let it try again up to a configurable amount of time. The reason
is that the fail-fast approach with the lower RPC timeout will continue to cause a failure
until the index region can be written to. This will allow us to ride out region moves without
a long RPC time out and thus without holding handler threads for long periods of time. We
can base the failure on the INDEX_DISABLE_TIMESTAMP value of an index as we walk through the
scan results here in MetaDataRegionObserver. :
> {code}
>                 do {
>                     results.clear();
>                     hasMore =;
>                     if (results.isEmpty()) break;
>                     Result r = Result.create(results);
>                     byte[] disabledTimeStamp = r.getValue(PhoenixDatabaseMetaData.TABLE_FAMILY_BYTES,
>                         PhoenixDatabaseMetaData.INDEX_DISABLE_TIMESTAMP_BYTES);
>                     byte[] indexState = r.getValue(PhoenixDatabaseMetaData.TABLE_FAMILY_BYTES,
>                             PhoenixDatabaseMetaData.INDEX_STATE_BYTES);
>                     if (disabledTimeStamp == null || disabledTimeStamp.length == 0) {
>                         continue;
>                     }
>                     // TODO: if disabledTimeStamp - System.currentTimeMillis() > configurableAmount

>                     // then disable the index.
> {code}
> I'd propose we allow 30 minutes to get an index back online.

This message was sent by Atlassian JIRA

View raw message