From issues-return-5663-archive-asf-public=cust-asf.ponee.io@phoenix.apache.org Thu Apr 4 20:07:03 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 7CFD618065B for ; Thu, 4 Apr 2019 22:07:02 +0200 (CEST) Received: (qmail 47190 invoked by uid 500); 4 Apr 2019 20:07:01 -0000 Mailing-List: contact issues-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list issues@phoenix.apache.org Received: (qmail 46908 invoked by uid 99); 4 Apr 2019 20:07:00 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Apr 2019 20:07:00 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 5D9F0E0708 for ; Thu, 4 Apr 2019 20:07:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 1F23424599 for ; Thu, 4 Apr 2019 20:07:00 +0000 (UTC) Date: Thu, 4 Apr 2019 20:07:00 +0000 (UTC) From: "Geoffrey Jacoby (JIRA)" To: issues@phoenix.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (PHOENIX-5194) Thread Cache is not update for Index retries in for MutationState#send()#doMutation() MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PHOENIX-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16810245#comment-16810245 ] Geoffrey Jacoby commented on PHOENIX-5194: ------------------------------------------ Change itself looks fine to me. Is there a way to have a test for this? > Thread Cache is not update for Index retries in for MutationState#send()#doMutation() > ------------------------------------------------------------------------------------- > > Key: PHOENIX-5194 > URL: https://issues.apache.org/jira/browse/PHOENIX-5194 > Project: Phoenix > Issue Type: Sub-task > Affects Versions: 4.14.0, 5.0.0, 4.15.0, 4.14.1 > Reporter: Monani Mihir > Assignee: Monani Mihir > Priority: Major > Labels: client > Attachments: PHOENIX-5194-4.x-HBase-1.3.01.patch, PHOENIX-5194-4.x-HBase-1.3.02.patch, PHOENIX-5194-4.x-HBase-1.3.03.patch, PHOENIX-5194-4.x-HBase-1.3.04.patch > > > Wwhen Client is writing and Index Failures happens, MutationState#send() will use PhoenixIndexFailurePolicy#doBatchWithRetries to apply index mutations. If during this retires Index region and Data table region moves , Index/Data table region location cache does not get updated. Because of this client is keep trying to write in same location and get failures. After all retries are finished, it will simply disable Index and aborts the client thread. > {noformat} > 2019-03-08 09:41:32,678 WARN [pool-8-thread-25] execute.MutationState - THREAD_ABORT MutationState#send(Iterator) :- > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 36 actions: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=1873403620592046670 region=PHERF:TABLE1,1552037797977.20beae29172b4bec422a6984e088eeae.host=phoenix-host1,60020,1552037496260 Index update failed > at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:112) > at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:86) > at org.apache.phoenix.index.PhoenixIndexMetaDataBuilder.getIndexMetaDataCache(PhoenixIndexMetaDataBuilder.java:101) > at org.apache.phoenix.index.PhoenixIndexMetaDataBuilder.getIndexMetaData(PhoenixIndexMetaDataBuilder.java:51) > at org.apache.phoenix.index.PhoenixIndexBuilder.getIndexMetaData(PhoenixIndexBuilder.java:100) > at org.apache.phoenix.index.PhoenixIndexBuilder.getIndexMetaData(PhoenixIndexBuilder.java:73) > at org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexMetaData(IndexBuildManager.java:79) > at org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:385) > at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:345) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:1025) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1693) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1771) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1727) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1021) > at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3309) > at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3076) > at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3018) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:914) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:842) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2397) > at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:35080) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2399) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) > Caused by: java.sql.SQLException: ERROR 2008 (INT10): Unable to find cached index metadata. key=1873403620592046670 region=PHERF:TABLE1,1552037797977.20beae29172b4bec422a6984e088eeae.host=phoenix-host1,60020,1552037496260 > at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:494) > at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150) > at org.apache.phoenix.index.PhoenixIndexMetaDataBuilder.getIndexMetaDataCache(PhoenixIndexMetaDataBuilder.java:100) > ... 22 more > : 36 times, servers with issues: phoenix-host1,60020,1552037496260 > at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:260) > at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$2400(AsyncProcess.java:240) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1711) > at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:917) > at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:931) > at org.apache.phoenix.execute.MutationState$3.doMutation(MutationState.java:992) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy.doBatchWithRetries(PhoenixIndexFailurePolicy.java:480) > at org.apache.phoenix.execute.MutationState.send(MutationState.java:988) > at org.apache.phoenix.execute.MutationState.send(MutationState.java:1368) > at org.apache.phoenix.execute.MutationState.commit(MutationState.java:1188) > at org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:670) > at org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:666) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at org.apache.phoenix.jdbc.PhoenixConnection.commit(PhoenixConnection.java:666) > at org.apache.phoenix.pherf.workload.WriteWorkload$2.call(WriteWorkload.java:297) > at org.apache.phoenix.pherf.workload.WriteWorkload$2.call(WriteWorkload.java:256) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)