From issues-return-6107-archive-asf-public=cust-asf.ponee.io@phoenix.apache.org Wed Apr 17 19:10:03 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9B09D180672 for ; Wed, 17 Apr 2019 21:10:02 +0200 (CEST) Received: (qmail 39599 invoked by uid 500); 17 Apr 2019 19:10:02 -0000 Mailing-List: contact issues-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list issues@phoenix.apache.org Received: (qmail 39590 invoked by uid 99); 17 Apr 2019 19:10:02 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Apr 2019 19:10:02 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id C3112E005C for ; Wed, 17 Apr 2019 19:10:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 75C2324595 for ; Wed, 17 Apr 2019 19:10:00 +0000 (UTC) Date: Wed, 17 Apr 2019 19:10:00 +0000 (UTC) From: "Alexander Saraseka (JIRA)" To: issues@phoenix.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (PHOENIX-4960) Write to table with global index failed if meta of index changed (split, move, etc) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PHOENIX-4960?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16= 820419#comment-16820419 ]=20 Alexander Saraseka commented on PHOENIX-4960: --------------------------------------------- Seems we have the same. Version 4.14.1-cdh5.14.2 - during the split (not a region of table used in = index), region server dies due to "Could not update the index table, killin= g server region because couldn't write to an index table" Split event started:=C2=A0 {code:java} 2019-04-04, 22:16:04.854 Transition null to {04bb34fa7f3f0696582d00ab4cf033= a6 state=3DSPLITTING_NEW, ts=3D1554405364851, server=3Dhbase-data003',60020= ,1553912711884} 2019-04-04, 22:16:04.854 Transition {386b310a7f88bbcf17b1f1d2af6e9a65 state= =3DOPEN, ts=3D1553912795381, server=3Dhbase-data003,60020,1553912711884} to= {386b310a7f88bbcf17b1f1d2af6e9a65 state=3DSPLITTING, ts=3D1554405364851, s= erver=3Dhbase-data003',60020,1553912711884} 2019-04-04, 22:16:04.853 Transition null to {6239476ebc6fefdd9d05917e1ea122= 62 state=3DSPLITTING_NEW, ts=3D1554405364851, server=3Dhbase-data003',60020= ,1553912711884} 2019-04-04, 22:16:04.840 Starting split of region SCHEMA_NAME:delivery_repo= rts_gatewayResponses,BRAND1\x00\xC4\x0FYc\x16\x005065412\x0013473599063\x00= d1acccb1af39ea36da8d3eaf9fc5fc35,1547818505037.386b310a7f88bbcf17b1f1d2af6e= 9a65. {code} Region server dies: {code} 2019-04-04, 22:16:05.065=09#339, the task was rejected by the pool. This is= unexpected. Server is hbase-data006,60020,1553912711613 2019-04-04, 22:16:05.069=09#339, table=3DSCHEMA_NAME:SCHEMA_NAME_DELIVERY_R= EPORTS_BRANDTYPE_ACCOUNTID_GLOBAL_IDX, attempt=3D1/1 failed=3D39ops, last e= xception: java.util.concurrent.RejectedExecutionException: Task java.util.c= oncurrent.FutureTask@7a7d7ea0 rejected from java.util.concurrent.ThreadPool= Executor@6730bfae[Terminated, pool size =3D 84, active threads =3D 0, queue= d tasks =3D 0, completed tasks =3D 1270825] on hbase-data006,60020,15539127= 11613, tracking started Thu Apr 04 19:16:05 UTC 2019; not retrying 39 - fin= al failure 2019-04-04, 22:16:05.157 Region server hbase-data003,60020,1553912711884 re= ported a fatal error: ABORTING region server hbase-data003,60020,1553912711884: Could not update = the index table, killing server region because couldn't write to an index t= able Cause: org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: = disableIndexOnFailure=3Dtrue, Failed to write to multiple index tables: [SC= HEMA_NAME:SCHEMA_NAME_DELIVERY_REPORTS_BRANDTYPE_ACCOUNTID_GLOBAL_IDX, SCHE= MA_NAME:SCHEMA_NAME_DELIVERY_REPORTS_BRANDTYPE_CAMPAIGNID_GLOBAL_IDX, SCHEM= A_NAME:SCHEMA_NAME_DELIVERY_REPORTS_BRANDTYPE_ACCOUNTID_FROMNUMBER_TONUMBER= _GLOBAL_IDX]{code} next time: Split event started:=C2=A0 {code}2019-04-15, 19:31:50.409=09Transition null to {67eb24356528b9b4ce6a24= 5a0e28b589 state=3DSPLITTING_NEW, ts=3D1555345910406, server=3Dhbase-data00= 1,60020,1555061890520} 2019-04-15, 19:31:50.393=09Starting split of region SCHEMA_NAME:delivery_re= ports_gatewayResponses,BRAND2\x00\xC6\x06\x06\x0D\x09Q\x04\x002870732177003= \x0018159781806\x000fd9ec224f7079ba0297e99056f8d583,1553532655360.a80f64680= 9cda80d56872cb899bacdbb. 2019-04-15, 19:31:50.411=09Transition null to {30bbf7c667e736e766d5ac3a1f32= 7f9f state=3DSPLITTING_NEW, ts=3D1555345910408, server=3Dhbase-data001,6002= 0,1555061890520} 2019-04-15, 19:31:50.411=09Transition {a80f646809cda80d56872cb899bacdbb sta= te=3DOPEN, ts=3D1555062460270, server=3Dhbase-data001,60020,1555061890520} = to {a80f646809cda80d56872cb899bacdbb state=3DSPLITTING, ts=3D1555345910408,= server=3Dhbase-data001,60020,1555061890520} {code} Region server dies: {code} 2019-04-15, 19:31:50.726=09Successfully update INDEX_DISABLE_TIMESTAMP for = SCHEMA_NAME:SCHEMA_NAME_DELIVERY_REPORTS_BRANDTYPE_ACCOUNTID_FROMNUMBER_TON= UMBER_GLOBAL_IDX due to an exception while writing updates. indexState=3DPE= NDING_DISABLE 2019-04-15, 19:31:50.732=09ABORTING region server hbase-data001,60020,15550= 61890520: Could not update the index table, killing server region because c= ouldn't write to an index table {code} > Write to table with global index failed if meta of index changed (split, = move, etc) > -------------------------------------------------------------------------= ---------- > > Key: PHOENIX-4960 > URL: https://issues.apache.org/jira/browse/PHOENIX-4960 > Project: Phoenix > Issue Type: Bug > Affects Versions: 4.14.0 > Reporter: Alex Batyrshin > Assignee: Vincent Poon > Priority: Critical > Fix For: 4.14.1 > > Attachments: PHOENIX-4960.v1.4.x-HBase-1.3.patch, PHOENIX-4960.v1= .master.patch, hbase-site-stage.xml > > > Tested on released 4.14.0 and on commit [https://github.com/apache/phoeni= x/commit/52893c240e4f24e2bfac0834d35205f866c16ed8] > Reproduce steps: > 1. Create table with global index > {code:sql} > 0: jdbc:phoenix:127.0.0.1> CREATE TABLE test_meta_change ( > . . . . . . . . . . . . .> pk VARCHAR NOT NULL PRIMARY KEY, > . . . . . . . . . . . . .> data VARCHAR > . . . . . . . . . . . . .> ); > No rows affected (1.359 seconds) > 0: jdbc:phoenix:127.0.0.1> CREATE INDEX test_meta_change_idx ON test_meta= _change (data); > No rows affected (6.333 seconds) > {code} > 2. Check that upsert works > {code:sql} > 0: jdbc:phoenix:127.0.0.1> UPSERT INTO test_meta_change VALUES ('a', 'foo= '); > 1 row affected (0.186 seconds) > {code} > 3. Move index region via HBase shell > {code:java} > hbase(main):005:0> move 'index-region-hash' > 0 row(s) in 0.1330 seconds > {code} > 4. Try to upsert again > {code:sql} > 0: jdbc:phoenix:127.0.0.1> UPSERT INTO test_meta_change VALUES ('b', 'bar= '); > 18/10/08 03:32:10 WARN client.AsyncProcess: #1, table=3DTEST_META_CHANGE,= attempt=3D1/35 failed=3D1ops, last exception: org.apache.hadoop.hbase.DoNo= tRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 112= 1 (XCL21): Write to the index failed. disableIndexOnFailure=3Dtrue, Failed > to write to multiple index tables: [TEST_META_CHANGE_IDX] ,serverTimestam= p=3D1538958729912, > at org.apache.phoenix.util.ServerUtil.wrapInDoNotRetryIOException= (ServerUtil.java:265) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailu= re(PhoenixIndexFailurePolicy.java:165) > at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillY= ourselfOnFailure(IndexWriter.java:161) > at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillY= ourselfOnFailure(IndexWriter.java:145) > at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(In= dexer.java:620) > at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:595= ) > at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispen= sably(Indexer.java:578) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.= call(RegionCoprocessorHost.java:1048) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$Reg= ionOperation.call(RegionCoprocessorHost.java:1711) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.exe= cOperation(RegionCoprocessorHost.java:1789) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.exe= cOperation(RegionCoprocessorHost.java:1745) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.pos= tBatchMutateIndispensably(RegionCoprocessorHost.java:1044) > at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutati= on(HRegion.java:3646) > at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegi= on.java:3108) > at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegi= on.java:3050) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(R= SRpcServices.java:916) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomic= RegionMutation(RSRpcServices.java:844) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpc= Services.java:2405) > at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$Client= Service$2.callBlockingMethod(ClientProtos.java:36621) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2359= ) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124= ) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecuto= r.java:297) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecuto= r.java:277) > Caused by: java.sql.SQLException: ERROR 1121 (XCL21): Write to the index = failed. disableIndexOnFailure=3Dtrue, Failed to write to multiple index ta= bles: [TEST_META_CHANGE_IDX] > at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newExc= eption(SQLExceptionCode.java:494) > at org.apache.phoenix.exception.SQLExceptionInfo.buildException(S= QLExceptionInfo.java:150) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailu= re(PhoenixIndexFailurePolicy.java:164) > ... 21 more > Caused by: org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailur= eException: disableIndexOnFailure=3Dtrue, Failed to write to multiple inde= x tables: [TEST_META_CHANGE_IDX] > at org.apache.phoenix.hbase.index.write.TrackingParallelWriterInd= exCommitter.write(TrackingParallelWriterIndexCommitter.java:236) > at org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWr= iter.java:195) > at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillY= ourselfOnFailure(IndexWriter.java:156) > ... 20 more > on stg-hbase5,60020,1538935600231, tracking started Mon Oct 08 03:32:09 = MSK 2018; not retrying 1 - final failure > 1 row affected (0.471 seconds) > {code} > 5. Retry > {code:java} > 0: jdbc:phoenix:127.0.0.1> UPSERT INTO test_meta_change VALUES ('b', 'bar= '); > Error: ERROR 1120 (XCL20): Writes to table blocked until index can be upd= ated. tableName=3DTEST_META_CHANGE (state=3DXCL20,code=3D1120) > java.sql.SQLException: ERROR 1120 (XCL20): Writes to table blocked until = index can be updated. tableName=3DTEST_META_CHANGE > at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newExc= eption(SQLExceptionCode.java:494) > at org.apache.phoenix.exception.SQLExceptionInfo.buildException(S= QLExceptionInfo.java:150) > at org.apache.phoenix.execute.MutationState.validateAndGetServerT= imestamp(MutationState.java:815) > at org.apache.phoenix.execute.MutationState.validateAll(MutationS= tate.java:789) > at org.apache.phoenix.execute.MutationState.send(MutationState.ja= va:981) > at org.apache.phoenix.execute.MutationState.send(MutationState.ja= va:1514) > at org.apache.phoenix.execute.MutationState.commit(MutationState.= java:1337) > at org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnec= tion.java:670) > at org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnec= tion.java:666) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at org.apache.phoenix.jdbc.PhoenixConnection.commit(PhoenixConnec= tion.java:666) > at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStateme= nt.java:411) > at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStateme= nt.java:391) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(Phoen= ixStatement.java:389) > at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(Phoen= ixStatement.java:378) > at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatem= ent.java:1825) > at sqlline.Commands.execute(Commands.java:822) > at sqlline.Commands.sql(Commands.java:732) > at sqlline.SqlLine.dispatch(SqlLine.java:813) > at sqlline.SqlLine.begin(SqlLine.java:686) > at sqlline.SqlLine.start(SqlLine.java:398) > at sqlline.SqlLine.main(SqlLine.java:291) > {code} > Logs from region server: > {code} > Oct 08 03:32:09 stg-hbase5 hbase[24993]: 2018-10-08 03:32:09,926 WARN [h= connection-0x49f74871-shared--pool12-t2] client.AsyncProcess: #66, table=3D= TEST_META_CHANGE_IDX, attempt=3D1/1 failed=3D1ops, last exception: org.apac= he.hadoop.hbase.exceptions.RegionMovedException: Region moved to: hostname= =3Dstg-hbase4 port=3D60020 startCode=3D1538935598023. As of locationSeqNum= =3D6. on stg-hbase5,60020,1538935600231, tracking started Mon Oct 08 03:32:= 09 MSK 2018; not retrying 1 - final failure > Oct 08 03:32:09 stg-hbase5 hbase[24993]: 2018-10-08 03:32:09,953 INFO [R= pcServer.default.FPBQ.Fifo.handler=3D98,queue=3D8,port=3D60020] zookeeper.R= ecoverableZooKeeper: Process identifier=3Dhconnection-0x6f725457 connecting= to ZooKeeper ensemble=3D10.73.66.21:2181,10.73.66.22:2181,10.73.66.181:218= 1 > Oct 08 03:32:09 stg-hbase5 hbase[24993]: 2018-10-08 03:32:09,953 INFO [R= pcServer.default.FPBQ.Fifo.handler=3D98,queue=3D8,port=3D60020] zookeeper.Z= ooKeeper: Initiating client connection, connectString=3D10.73.66.21:2181,10= .73.66.22:2181,10.73.66.181:2181 sessionTimeout=3D90000 watcher=3Dorg.apach= e.hadoop.hbase.zookeeper.PendingWatcher@dcf4b2 > Oct 08 03:32:09 stg-hbase5 hbase[24993]: 2018-10-08 03:32:09,957 INFO [R= pcServer.default.FPBQ.Fifo.handler=3D98,queue=3D8,port=3D60020-SendThread(1= 0.73.66.22:2181)] zookeeper.ClientCnxn: Opening socket connection to server= 10.73.66.22/10.73.66.22:2181. Will not attempt to authenticate using SASL = (unknown error) > Oct 08 03:32:09 stg-hbase5 hbase[24993]: 2018-10-08 03:32:09,959 INFO [R= pcServer.default.FPBQ.Fifo.handler=3D98,queue=3D8,port=3D60020-SendThread(1= 0.73.66.22:2181)] zookeeper.ClientCnxn: Socket connection established to 10= .73.66.22/10.73.66.22:2181, initiating session > Oct 08 03:32:09 stg-hbase5 hbase[24993]: 2018-10-08 03:32:09,962 INFO [R= pcServer.default.FPBQ.Fifo.handler=3D98,queue=3D8,port=3D60020-SendThread(1= 0.73.66.22:2181)] zookeeper.ClientCnxn: Session establishment complete on s= erver 10.73.66.22/10.73.66.22:2181, sessionid =3D 0x20009a64ef6a15c, negoti= ated timeout =3D 40000 > Oct 08 03:32:10 stg-hbase5 hbase[24993]: 2018-10-08 03:32:09,997 INFO [R= pcServer.default.FPBQ.Fifo.handler=3D98,queue=3D8,port=3D60020] index.Phoen= ixIndexFailurePolicy: Successfully update INDEX_DISABLE_TIMESTAMP for TEST_= META_CHANGE_IDX due to an exception while writing updates. indexState=3DPEN= DING_DISABLE > Oct 08 03:32:10 stg-hbase5 hbase[24993]: org.apache.phoenix.hbase.index.e= xception.MultiIndexWriteFailureException: disableIndexOnFailure=3Dtrue, Fa= iled to write to multiple index tables: [TEST_META_CHANGE_IDX] > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.phoenix.hb= ase.index.write.TrackingParallelWriterIndexCommitter.write(TrackingParallel= WriterIndexCommitter.java:236) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.phoenix.hb= ase.index.write.IndexWriter.write(IndexWriter.java:195) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.phoenix.hb= ase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:= 156) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.phoenix.hb= ase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:= 145) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.phoenix.hb= ase.index.Indexer.doPostWithExceptions(Indexer.java:620) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.phoenix.hb= ase.index.Indexer.doPost(Indexer.java:595) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.phoenix.hb= ase.index.Indexer.postBatchMutateIndispensably(Indexer.java:578) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:10= 48) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocesso= rHost.java:1711) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.j= ava:1789) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.j= ava:1745) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCo= processorHost.java:1044) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3646) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.regionserver.HRegion.batchMutate(HRegion.java:3108) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.regionserver.HRegion.batchMutate(HRegion.java:3050) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:916) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:= 844) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.regionserver.RSRpcServices.multi(RSRpcServices.java:2405) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(Clien= tProtos.java:36621) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.ipc.RpcServer.call(RpcServer.java:2359) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.ipc.CallRunner.run(CallRunner.java:124) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: at org.apache.hadoop.hba= se.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) > Oct 08 03:32:10 stg-hbase5 hbase[24993]: 2018-10-08 03:32:10,003 INFO [R= pcServer.default.FPBQ.Fifo.handler=3D98,queue=3D8,port=3D60020] util.IndexM= anagementUtil: Rethrowing org.apache.hadoop.hbase.DoNotRetryIOException: ER= ROR 1121 (XCL21): Write to the index failed. disableIndexOnFailure=3Dtrue,= Failed to write to multiple index tables: [TEST_META_CHANGE_IDX] ,serverTi= mestamp=3D1538958729912, > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)