From dev-return-50706-archive-asf-public=cust-asf.ponee.io@phoenix.apache.org Mon Apr 9 16:55:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id DC811180718 for ; Mon, 9 Apr 2018 16:55:04 +0200 (CEST) Received: (qmail 48466 invoked by uid 500); 9 Apr 2018 14:55:03 -0000 Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list dev@phoenix.apache.org Received: (qmail 48455 invoked by uid 99); 9 Apr 2018 14:55:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Apr 2018 14:55:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 7D08318061F for ; Mon, 9 Apr 2018 14:55:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.511 X-Spam-Level: X-Spam-Status: No, score=-109.511 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id wUEUP4huXdXM for ; Mon, 9 Apr 2018 14:55:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 2BB7E5FB37 for ; Mon, 9 Apr 2018 14:55:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 67A26E09A4 for ; Mon, 9 Apr 2018 14:55:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 1EB9B241C6 for ; Mon, 9 Apr 2018 14:55:00 +0000 (UTC) Date: Mon, 9 Apr 2018 14:55:00 +0000 (UTC) From: "Rajeshbabu Chintaguntla (JIRA)" To: dev@phoenix.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (PHOENIX-4685) Parallel writes continuously to indexed table failing with OOME very quickly in 5.x branch MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PHOENIX-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajeshbabu Chintaguntla updated PHOENIX-4685: --------------------------------------------- Description: Currently trying to write data to indexed table failing with OOME where unable to create native threads. But it's working fine with 4.7.x branches. Found many threads created for meta lookup and shared threads and no space to create threads. {noformat} 2018-04-08 13:06:04,747 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=0,port=16020] index.PhoenixIndexFailurePolicy: handleFailure failed java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:185) at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailureWithExceptions(PhoenixIndexFailurePolicy.java:217) at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:143) at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:160) at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144) at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:632) at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:607) at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:590) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1037) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1034) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1034) at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:3533) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3914) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3822) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3753) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Caused by: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761) at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448) at org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429) at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:40) at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:183) ... 25 more Caused by: java.lang.Exception: java.lang.OutOfMemoryError: unable to create new native thread at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:266) at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:217) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) ... 32 more Caused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1007) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:986) at org.apache.phoenix.util.IndexUtil.updateIndexState(IndexUtil.java:724) at org.apache.phoenix.util.IndexUtil.updateIndexState(IndexUtil.java:709) at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:236) ... 36 more {noformat} was: Currently trying to write data to indexed table failing with OOME where unable to create native threads. But it's working fine with 4.7.x branches. Found many threads created for meta lookup and shared threads and no space to create threads. > Parallel writes continuously to indexed table failing with OOME very quickly in 5.x branch > ------------------------------------------------------------------------------------------ > > Key: PHOENIX-4685 > URL: https://issues.apache.org/jira/browse/PHOENIX-4685 > Project: Phoenix > Issue Type: Bug > Reporter: Rajeshbabu Chintaguntla > Assignee: Rajeshbabu Chintaguntla > Priority: Major > Fix For: 5.0.0 > > Attachments: PHOENIX-4685_jstack > > > Currently trying to write data to indexed table failing with OOME where unable to create native threads. But it's working fine with 4.7.x branches. Found many threads created for meta lookup and shared threads and no space to create threads. > {noformat} > 2018-04-08 13:06:04,747 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=0,port=16020] index.PhoenixIndexFailurePolicy: handleFailure failed > java.io.IOException: java.lang.reflect.UndeclaredThrowableException > at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:185) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailureWithExceptions(PhoenixIndexFailurePolicy.java:217) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy.handleFailure(PhoenixIndexFailurePolicy.java:143) > at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:160) > at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144) > at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:632) > at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:607) > at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:590) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1037) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$30.call(RegionCoprocessorHost.java:1034) > at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540) > at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1034) > at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.doPostOpCleanupForMiniBatch(HRegion.java:3533) > at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3914) > at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3822) > at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3753) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1027) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:959) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:922) > at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2666) > at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) > Caused by: java.lang.reflect.UndeclaredThrowableException > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761) > at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448) > at org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429) > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at org.apache.hadoop.hbase.util.Methods.call(Methods.java:40) > at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:183) > ... 25 more > Caused by: java.lang.Exception: java.lang.OutOfMemoryError: unable to create new native thread > at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:266) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:217) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) > ... 32 more > Caused by: java.lang.OutOfMemoryError: unable to create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:714) > at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) > at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357) > at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) > at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1007) > at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:986) > at org.apache.phoenix.util.IndexUtil.updateIndexState(IndexUtil.java:724) > at org.apache.phoenix.util.IndexUtil.updateIndexState(IndexUtil.java:709) > at org.apache.phoenix.index.PhoenixIndexFailurePolicy$1.run(PhoenixIndexFailurePolicy.java:236) > ... 36 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)