phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "cuizhaohua (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PHOENIX-5241) Write to table with global index failed if meta of index changed (split, move, etc)
Date Mon, 15 Apr 2019 07:33:00 GMT

     [ https://issues.apache.org/jira/browse/PHOENIX-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

cuizhaohua updated PHOENIX-5241:
--------------------------------
    Description: 
HBase version :1.2.6

phoenix version: phoenix-4.14.1-HBase-1.2  (download from [http://phoenix.apache.org/download.html])

phoneinx client version:  phoenix-4.14.1-HBase-1.2   (download from [http://phoenix.apache.org/download.html])

step 1:

0: jdbc:phoenix:xxxx:/hbase> UPSERT INTO test_meta_change VALUES ('1', 'foo');
 1 row affected (0.298 seconds)

 

setp 2: move index region 

hbase(main):008:0> move '0b158edd48c60560c358a3208fee8e24'
 0 row(s) in 0.0500 seconds

 

step 3: get the error

0: jdbc:phoenix:xxxx:/hbase> UPSERT INTO test_meta_change VALUES ('2', 'foo');
 19/04/15 15:12:29 WARN client.AsyncProcess: #1, table=TEST_META_CHANGE, attempt=1/35 failed=1ops,
last exception: org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException:
ERROR 1121 (XCL21): Write to the index failed. disableIndexOnFailure=true, Failed to write
to multiple index tables: [TEST_META_CHANGE_IDX] ,serverTimestamp=1555312349291,
 at org.apache.phoenix.util.ServerUtil.wrapInDoNotRetryIOException(ServerUtil.java:265)
 at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:172)
 at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:161)
 at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:145)
 at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:623)
 at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:583)
 at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:566)
 at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1034)
 at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
 at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
 at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
 at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1030)
 at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3324)
 at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2881)
 at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2823)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:758)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:720)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2168)
 at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
 at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
 at java.lang.Thread.run(Thread.java:745)
 Caused by: java.sql.SQLException: ERROR 1121 (XCL21): Write to the index failed. disableIndexOnFailure=true,
Failed to write to multiple index tables: [TEST_META_CHANGE_IDX]
 at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:494)
 at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150)
 at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:171)
 ... 22 more
 Caused by: org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: disableIndexOnFailure=true,
Failed to write to multiple index tables: [TEST_META_CHANGE_IDX]
 at org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:236)
 at org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:195)
 at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:156)
 ... 21 more
 on hbasefat031219,16020,1555307435151, tracking started Mon Apr 15 15:12:29 CST 2019; not
retrying 1 - final failure
 1 row affected (0.625 seconds)

 

 

PS: if I download apache-phoenix-4.13.1-HBase-1.2-bin  as phoenix client , It works fine
.

 

It's like phoenix-4960 :

https://issues.apache.org/jira/browse/PHOENIX-4960

 

Thanks  for any help!

  was:
HBase version :1.2.6

phoenix version: phoenix-4.14.1-HBase-1.2  (download from [http://phoenix.apache.org/download.html])

phoneinx client version:  phoenix-4.14.1-HBase-1.2   (download from [http://phoenix.apache.org/download.html])

step 1:

0: jdbc:phoenix:xxxx:/hbase> UPSERT INTO test_meta_change VALUES ('1', 'foo');
1 row affected (0.298 seconds)

 

setp 2: move index region 

hbase(main):008:0> move '0b158edd48c60560c358a3208fee8e24'
0 row(s) in 0.0500 seconds

 

step 3: get the error

0: jdbc:phoenix:xxxx:/hbase> UPSERT INTO test_meta_change VALUES ('2', 'foo');
19/04/15 15:12:29 WARN client.AsyncProcess: #1, table=TEST_META_CHANGE, attempt=1/35 failed=1ops,
last exception: org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException:
ERROR 1121 (XCL21): Write to the index failed. disableIndexOnFailure=true, Failed to write
to multiple index tables: [TEST_META_CHANGE_IDX] ,serverTimestamp=1555312349291,
 at org.apache.phoenix.util.ServerUtil.wrapInDoNotRetryIOException(ServerUtil.java:265)
 at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:172)
 at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:161)
 at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:145)
 at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:623)
 at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:583)
 at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:566)
 at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1034)
 at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
 at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
 at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
 at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1030)
 at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3324)
 at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2881)
 at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2823)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:758)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:720)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2168)
 at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
 at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: ERROR 1121 (XCL21): Write to the index failed. disableIndexOnFailure=true,
Failed to write to multiple index tables: [TEST_META_CHANGE_IDX]
 at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:494)
 at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150)
 at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:171)
 ... 22 more
Caused by: org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: disableIndexOnFailure=true,
Failed to write to multiple index tables: [TEST_META_CHANGE_IDX]
 at org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:236)
 at org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:195)
 at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:156)
 ... 21 more
 on hbasefat031219,16020,1555307435151, tracking started Mon Apr 15 15:12:29 CST 2019; not
retrying 1 - final failure
1 row affected (0.625 seconds)

 

 

PS: if I download apache-phoenix-4.13.1-HBase-1.2-bin  as phoenix client , It works fine
.

 

Thanks  for any help!


> Write to table with global index failed if meta of index changed (split, move, etc)
> -----------------------------------------------------------------------------------
>
>                 Key: PHOENIX-5241
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-5241
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.14.1
>         Environment: phoenix-4.14.1-HBase-1.2
>            Reporter: cuizhaohua
>            Priority: Major
>
> HBase version :1.2.6
> phoenix version: phoenix-4.14.1-HBase-1.2  (download from [http://phoenix.apache.org/download.html])
> phoneinx client version:  phoenix-4.14.1-HBase-1.2   (download from [http://phoenix.apache.org/download.html])
> step 1:
> 0: jdbc:phoenix:xxxx:/hbase> UPSERT INTO test_meta_change VALUES ('1', 'foo');
>  1 row affected (0.298 seconds)
>  
> setp 2: move index region 
> hbase(main):008:0> move '0b158edd48c60560c358a3208fee8e24'
>  0 row(s) in 0.0500 seconds
>  
> step 3: get the error
> 0: jdbc:phoenix:xxxx:/hbase> UPSERT INTO test_meta_change VALUES ('2', 'foo');
>  19/04/15 15:12:29 WARN client.AsyncProcess: #1, table=TEST_META_CHANGE, attempt=1/35
failed=1ops, last exception: org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException:
ERROR 1121 (XCL21): Write to the index failed. disableIndexOnFailure=true, Failed to write
to multiple index tables: [TEST_META_CHANGE_IDX] ,serverTimestamp=1555312349291,
>  at org.apache.phoenix.util.ServerUtil.wrapInDoNotRetryIOException(ServerUtil.java:265)
>  at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:172)
>  at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:161)
>  at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:145)
>  at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:623)
>  at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:583)
>  at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:566)
>  at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1034)
>  at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
>  at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
>  at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
>  at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1030)
>  at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3324)
>  at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2881)
>  at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2823)
>  at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:758)
>  at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:720)
>  at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2168)
>  at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
>  at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)
>  at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
>  at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
>  at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
>  at java.lang.Thread.run(Thread.java:745)
>  Caused by: java.sql.SQLException: ERROR 1121 (XCL21): Write to the index failed. disableIndexOnFailure=true,
Failed to write to multiple index tables: [TEST_META_CHANGE_IDX]
>  at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:494)
>  at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150)
>  at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:171)
>  ... 22 more
>  Caused by: org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException:
disableIndexOnFailure=true, Failed to write to multiple index tables: [TEST_META_CHANGE_IDX]
>  at org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:236)
>  at org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:195)
>  at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:156)
>  ... 21 more
>  on hbasefat031219,16020,1555307435151, tracking started Mon Apr 15 15:12:29 CST 2019;
not retrying 1 - final failure
>  1 row affected (0.625 seconds)
>  
>  
> PS: if I download apache-phoenix-4.13.1-HBase-1.2-bin  as phoenix client , It works
fine .
>  
> It's like phoenix-4960 :
> https://issues.apache.org/jira/browse/PHOENIX-4960
>  
> Thanks  for any help!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message