From issues-return-5518-archive-asf-public=cust-asf.ponee.io@phoenix.apache.org Wed Mar 27 12:07:02 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5DE6B180648 for ; Wed, 27 Mar 2019 13:07:02 +0100 (CET) Received: (qmail 408 invoked by uid 500); 27 Mar 2019 12:07:01 -0000 Mailing-List: contact issues-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list issues@phoenix.apache.org Received: (qmail 393 invoked by uid 99); 27 Mar 2019 12:07:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Mar 2019 12:07:01 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id EF05BE015C for ; Wed, 27 Mar 2019 12:07:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id B4E8624421 for ; Wed, 27 Mar 2019 12:07:00 +0000 (UTC) Date: Wed, 27 Mar 2019 12:07:00 +0000 (UTC) From: "Monani Mihir (JIRA)" To: issues@phoenix.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (PHOENIX-5194) Thread Cache is not update for Index retries in for MutationState#send()#doMutation() MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PHOENIX-5194?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16= 802704#comment-16802704 ]=20 Monani Mihir commented on PHOENIX-5194: --------------------------------------- Out of 5 IT failures, two are unrelated (=C2=A0NonColumnEncodedImmutableTxS= tatsCollectorIT and=C2=A0PhoenixTracingEndToEndIT) . For other three ITs, they are related. I am working on understanding and mo= difying these three ITs > Thread Cache is not update for Index retries in for MutationState#send()#= doMutation() > -------------------------------------------------------------------------= ------------ > > Key: PHOENIX-5194 > URL: https://issues.apache.org/jira/browse/PHOENIX-5194 > Project: Phoenix > Issue Type: Sub-task > Affects Versions: 4.14.0, 5.0.0, 4.15.0, 4.14.1 > Reporter: Monani Mihir > Assignee: Monani Mihir > Priority: Major > Labels: client > Attachments: PHOENIX-5194-4.x-HBase-1.3.01.patch, PHOENIX-5194-4.= x-HBase-1.3.02.patch > > > Wwhen Client is writing and Index Failures happens, MutationState#send() = will use PhoenixIndexFailurePolicy#doBatchWithRetries to apply index mutati= ons. If during this retires Index region and Data table region moves , Inde= x/Data table region location cache does not get updated. Because of this cl= ient is keep trying to write in same location and get failures. After all r= etries are finished, it will simply disable Index and aborts the client thr= ead. > {noformat} > 2019-03-08 09:41:32,678 WARN [pool-8-thread-25] execute.MutationState - T= HREAD_ABORT MutationState#send(Iterator) :- > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Fail= ed 36 actions: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (I= NT10): ERROR 2008 (INT10): Unable to find cached index metadata. key=3D1873= 403620592046670 region=3DPHERF:TABLE1,1552037797977.20beae29172b4bec422a698= 4e088eeae.host=3Dphoenix-host1,60020,1552037496260 Index update failed > at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:1= 12) > at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:86= ) > at org.apache.phoenix.index.PhoenixIndexMetaDataBuilder.getIndexMetaDataC= ache(PhoenixIndexMetaDataBuilder.java:101) > at org.apache.phoenix.index.PhoenixIndexMetaDataBuilder.getIndexMetaData(= PhoenixIndexMetaDataBuilder.java:51) > at org.apache.phoenix.index.PhoenixIndexBuilder.getIndexMetaData(PhoenixI= ndexBuilder.java:100) > at org.apache.phoenix.index.PhoenixIndexBuilder.getIndexMetaData(PhoenixI= ndexBuilder.java:73) > at org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexMetaD= ata(IndexBuildManager.java:79) > at org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(In= dexer.java:385) > at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:345= ) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(Reg= ionCoprocessorHost.java:1025) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOpera= tion.call(RegionCoprocessorHost.java:1693) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperati= on(RegionCoprocessorHost.java:1771) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperati= on(RegionCoprocessorHost.java:1727) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMut= ate(RegionCoprocessorHost.java:1021) > at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegi= on.java:3309) > at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:= 3076) > at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:= 3018) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServ= ices.java:914) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMu= tation(RSRpcServices.java:842) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices= .java:2397) > at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$= 2.callBlockingMethod(ClientProtos.java:35080) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2399) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:1= 88) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:1= 68) > Caused by: java.sql.SQLException: ERROR 2008 (INT10): Unable to find cach= ed index metadata. key=3D1873403620592046670 region=3DPHERF:TABLE1,15520377= 97977.20beae29172b4bec422a6984e088eeae.host=3Dphoenix-host1,60020,155203749= 6260 > at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(S= QLExceptionCode.java:494) > at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExcept= ionInfo.java:150) > at org.apache.phoenix.index.PhoenixIndexMetaDataBuilder.getIndexMetaDataC= ache(PhoenixIndexMetaDataBuilder.java:100) > ... 22 more > : 36 times, servers with issues: phoenix-host1,60020,1552037496260 > at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(= AsyncProcess.java:260) > at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$2400(As= yncProcess.java:240) > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.get= Errors(AsyncProcess.java:1711) > at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:917) > at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:931) > at org.apache.phoenix.execute.MutationState$3.doMutation(MutationState.ja= va:992) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy.doBatchWithRetries(= PhoenixIndexFailurePolicy.java:480) > at org.apache.phoenix.execute.MutationState.send(MutationState.java:988) > at org.apache.phoenix.execute.MutationState.send(MutationState.java:1368) > at org.apache.phoenix.execute.MutationState.commit(MutationState.java:118= 8) > at org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.jav= a:670) > at org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.jav= a:666) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at org.apache.phoenix.jdbc.PhoenixConnection.commit(PhoenixConnection.jav= a:666) > at org.apache.phoenix.pherf.workload.WriteWorkload$2.call(WriteWorkload.j= ava:297) > at org.apache.phoenix.pherf.workload.WriteWorkload$2.call(WriteWorkload.j= ava:256) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j= ava:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.= java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)