hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-19851) ProcV2 framework's rollback() needs to maintain stack like execute()
Date Thu, 15 Feb 2018 21:20:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366272#comment-16366272
] 

stack commented on HBASE-19851:
-------------------------------

I saw another version of this issue in nightly. We have reached one of the states below:

{code}
      case MERGE_TABLE_REGIONS_POST_OPERATION:
      case MERGE_TABLE_REGIONS_OPEN_MERGED_REGION:
      case MERGE_TABLE_REGIONS_POST_MERGE_COMMIT_OPERATION:
      case MERGE_TABLE_REGIONS_UPDATE_META:
        String msg = this + " We are in the " + state + " state."
            + " It is complicated to rollback the merge operation that region server is working
on."
            + " Rollback is not supported and we should let the merge operation to complete";
        LOG.warn(msg);
        // PONR
        throw new UnsupportedOperationException(this + " unhandled state=" + state);
{code}

We throw the UnsupportedOperationException which then tickles a CODE-BUG warning in the framework
which is its way of saying not-supposed-to-happen and it is right; the merge procedure is
not complete.



{code}
2018-02-15 16:18:58,492 WARN  [PEWorker-1] assignment.MergeTableRegionsProcedure(311): Failed
rollback attempt step MERGE_TABLE_REGIONS_UPDATE_META for merging the regions [dcd17e940fbb453cf6df1cee87b0cec7,
2b03bbfd418c9b3533bc2bb3cf6ce748] in table testRollbackAndDoubleExecution
java.lang.UnsupportedOperationException: pid=46, state=FAILED:MERGE_TABLE_REGIONS_UPDATE_META,
exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException:
abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[dcd17e940fbb453cf6df1cee87b0cec7,
2b03bbfd418c9b3533bc2bb3cf6ce748], forcibly=false unhandled state=MERGE_TABLE_REGIONS_UPDATE_META
  at org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.rollbackState(MergeTableRegionsProcedure.java:291)
  at org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.rollbackState(MergeTableRegionsProcedure.java:78)
  at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:199)
  at org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:859)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1356)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1312)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1181)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
2018-02-15 16:18:58,497 ERROR [PEWorker-1] helpers.MarkerIgnoringBase(159): CODE-BUG: Uncaught
runtime exception for pid=46, state=FAILED:MERGE_TABLE_REGIONS_UPDATE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException
via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException:
abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[dcd17e940fbb453cf6df1cee87b0cec7,
2b03bbfd418c9b3533bc2bb3cf6ce748], forcibly=false
java.lang.UnsupportedOperationException: pid=46, state=FAILED:MERGE_TABLE_REGIONS_UPDATE_META,
exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException:
abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[dcd17e940fbb453cf6df1cee87b0cec7,
2b03bbfd418c9b3533bc2bb3cf6ce748], forcibly=false unhandled state=MERGE_TABLE_REGIONS_UPDATE_META
  at org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.rollbackState(MergeTableRegionsProcedure.java:291)
  at org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.rollbackState(MergeTableRegionsProcedure.java:78)
  at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:199)
  at org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:859)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1356)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1312)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1181)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
{code}


> ProcV2 framework's rollback() needs to maintain stack like execute()
> --------------------------------------------------------------------
>
>                 Key: HBASE-19851
>                 URL: https://issues.apache.org/jira/browse/HBASE-19851
>             Project: HBase
>          Issue Type: Improvement
>          Components: proc-v2
>    Affects Versions: 2.0.0-beta-1
>            Reporter: Umesh Agashe
>            Priority: Major
>             Fix For: 2.0.0
>
>
> Refer to HBASE-19839 and detailed description there. Bug showed up as a flaky test TestMergeTableRegionsProcedure.testRollbackAndDoubleExecution().
In short, when rollback() is called after step 5 (MERGE_TABLE_REGIONS_CLOSE_REGIONS), rollback
of the procedure tries to reassign two regions that are to be merged. It spawns 2 AssignProcedures
and don't wait on them to finish. Rollback should have ability to have sub-procedures and
separate proc stack.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message