From dev-return-49628-archive-asf-public=cust-asf.ponee.io@phoenix.apache.org Thu Feb 22 06:04:12 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id BB81B18061A for ; Thu, 22 Feb 2018 06:04:11 +0100 (CET) Received: (qmail 5073 invoked by uid 500); 22 Feb 2018 05:04:10 -0000 Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list dev@phoenix.apache.org Received: (qmail 5062 invoked by uid 99); 22 Feb 2018 05:04:10 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Feb 2018 05:04:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 26FDAC0147 for ; Thu, 22 Feb 2018 05:04:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.511 X-Spam-Level: X-Spam-Status: No, score=-109.511 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id SM2tnJyWOytX for ; Thu, 22 Feb 2018 05:04:07 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id B3E9A5F397 for ; Thu, 22 Feb 2018 05:04:06 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 0BA62E01AD for ; Thu, 22 Feb 2018 05:04:04 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 9F69421001 for ; Thu, 22 Feb 2018 05:04:02 +0000 (UTC) Date: Thu, 22 Feb 2018 05:04:02 +0000 (UTC) From: "Vikas Vishwakarma (JIRA)" To: dev@phoenix.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (PHOENIX-4625) memory leak in PhoenixConnection if scanner renew lease thread is not enabled MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PHOENIX-4625?page=3Dcom.atlass= ian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Vishwakarma updated PHOENIX-4625: --------------------------------------- Description:=20 We have two different code path # In=C2=A0ConnectionQueryServicesImpl=C2=A0RenewLeaseTasks is scheduled ba= sed on the following checks=C2=A0=C2=A0if renew lease feature is supported = and if the renew lease config is enabled supportsFeature(ConnectionQuerySer= vices.Feature.RENEW_LEASE) && renewLeaseEnabled # In=C2=A0PhoenixConnection for every scan iterator is added to a Queue fo= r lease renewal based on just the check if the renew lease feature is suppo= rted=C2=A0services.supportsFeature(Feature.RENEW_LEASE) In PhoenixConnection we however miss the check whether renew lease config i= s enabled (phoenix.scanner.lease.renew.enabled) =C2=A0 Now consider a situation where Renew lease feature is supported but=C2=A0ph= oenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In th= is case PhoenixConnection will keep adding the iterators for every scan int= o the=C2=A0scannerQueue for renewal based on the feature supported check bu= t the renewal task is not running because=C2=A0phoenix.scanner.lease.renew.= enabled is set to false, so the=C2=A0scannerQueue will keep growing as long= as the PhoenixConnection is alive and multiple scans requests are coming o= n this connection. =C2=A0 We have a use case that uses a single PhoenixConnection that is perpetual a= nd does billions of scans on this connection. In this case=C2=A0scannerQueu= e is growing to several GB's and ultimately leading to Consecutive Full GC'= s/OOM =C2=A0 Add iterators for Lease renewal in=C2=A0PhoenixConnection =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D {code:java} =C2=A0 public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) { if (services.supportsFeature(Feature.RENEW_LEASE)) {=20 checkNotNull(itr); scannerQueue.add(new WeakReference(itr));=20 } } {code} =C2=A0 Starting the=C2=A0RenewLeaseTask =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D checks if=C2=A0Feature.RENEW_LEASE is supported and if phoenix.scanner.leas= e.renew.enabled is true and starts the=C2=A0RenewLeaseTask {code:java} =C2=A0 ConnectionQueryServicesImpl { .... this.renewLeaseEnabled =3D config.getBoolean(RENEW_LEASE_ENABLED, DEFAULT_R= ENEW_LEASE_ENABLED); ..... @Override public boolean isRenewingLeasesEnabled(){=20 return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && r= enewLeaseEnabled;=20 } private void scheduleRenewLeaseTasks() { if (isRenewingLeasesEnabled()) { renewLeaseExecutor =3D Executors.newScheduledThreadPool(renewLeasePoolSize, renewLeaseThreadFac= tory); for (LinkedBlockingQueue> q : connectio= nQueues) {=20 renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, renew= LeaseTaskFrequency, TimeUnit.MILLISECONDS);=20 } } } ....... } {code} =C2=A0 To solve this We must add both checks in PhoenixConnection if the feature i= s supported and if the config is enabled before adding the iterators to sca= nnerQueue ConnectionQueryServices.Feature.RENEW_LEASE is true=C2=A0 &&=C2=A0 phoenix.= scanner.lease.renew.enabled is true=C2=A0 instead of just checking if the feature=C2=A0ConnectionQueryServices.Featur= e.RENEW_LEASE is supported =C2=A0 =C2=A0 was: We have two different code path # In=C2=A0ConnectionQueryServicesImpl=C2=A0RenewLeaseTasks is scheduled ba= sed on the following checks=C2=A0=C2=A0if renew lease feature is supported = and if the renew lease config is enabled supportsFeature(ConnectionQuerySer= vices.Feature.RENEW_LEASE) && renewLeaseEnabled # In=C2=A0PhoenixConnection for every scan iterator is added to a Queue fo= r lease renewal based on just the check if the renew lease feature is suppo= rted=C2=A0services.supportsFeature(Feature.RENEW_LEASE) In PhoenixConnection we however miss the check whether renew lease config i= s enabled (phoenix.scanner.lease.renew.enabled) =C2=A0 Now consider a situation where Renew lease feature is supported but=C2=A0ph= oenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In th= is case PhoenixConnection will keep adding the iterators for every scan int= o the=C2=A0scannerQueue for renewal based on the feature supported check bu= t the renewal task is not running because=C2=A0phoenix.scanner.lease.renew.= enabled is set to false, so the=C2=A0scannerQueue will keep growing as long= as the PhoenixConnection is alive and multiple scans requests are coming o= n this connection. =C2=A0 We have a use case that uses a single PhoenixConnection that is perpetual a= nd does billions of scans on this connection. In this case=C2=A0scannerQueu= e is growing to several GB's and ultimately leading to Consecutive Full GC'= s/OOM =C2=A0 Add iterators for Lease renewal in=C2=A0PhoenixConnection =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D {code}=C2=A0 public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) { if (services.supportsFeature(Feature.RENEW_LEASE)) { checkNotNull(itr); scannerQueue.add(new WeakReference(itr)); } } {code} =C2=A0 Starting the=C2=A0RenewLeaseTask =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D checks if=C2=A0Feature.RENEW_LEASE is supported and if phoenix.scanner.leas= e.renew.enabled is true and starts the=C2=A0RenewLeaseTask {code}=C2=A0 ConnectionQueryServicesImpl { .... this.renewLeaseEnabled =3D config.getBoolean(RENEW_LEASE_ENABLED, DEFAULT_R= ENEW_LEASE_ENABLED); ..... @Override public boolean isRenewingLeasesEnabled() { return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && re= newLeaseEnabled; } private void scheduleRenewLeaseTasks() { if (isRenewingLeasesEnabled()) { renewLeaseExecutor =3D Executors.newScheduledThreadPool(renewLeasePoolSize, renewLeaseThreadFacto= ry); for (LinkedBlockingQueue> q : connectionQ= ueues) { renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, renewLea= seTaskFrequency, TimeUnit.MILLISECONDS); } } } ....... } {code}=C2=A0 To solve this We must add both checks in PhoenixConnection if the feature i= s supported and if the config is enabled before adding the iterators to sca= nnerQueue ConnectionQueryServices.Feature.RENEW_LEASE is true=C2=A0 &&=C2=A0 phoenix.= scanner.lease.renew.enabled is true=C2=A0 instead of just checking if the feature=C2=A0ConnectionQueryServices.Featur= e.RENEW_LEASE is supported =C2=A0 =C2=A0 > memory leak in PhoenixConnection if scanner renew lease thread is not ena= bled > -------------------------------------------------------------------------= ---- > > Key: PHOENIX-4625 > URL: https://issues.apache.org/jira/browse/PHOENIX-4625 > Project: Phoenix > Issue Type: Bug > Affects Versions: 4.13.0 > Reporter: Vikas Vishwakarma > Priority: Major > Attachments: QS.png > > > We have two different code path > # In=C2=A0ConnectionQueryServicesImpl=C2=A0RenewLeaseTasks is scheduled = based on the following checks=C2=A0=C2=A0if renew lease feature is supporte= d and if the renew lease config is enabled supportsFeature(ConnectionQueryS= ervices.Feature.RENEW_LEASE) && renewLeaseEnabled > # In=C2=A0PhoenixConnection for every scan iterator is added to a Queue = for lease renewal based on just the check if the renew lease feature is sup= ported=C2=A0services.supportsFeature(Feature.RENEW_LEASE) > In PhoenixConnection we however miss the check whether renew lease config= is enabled (phoenix.scanner.lease.renew.enabled) > =C2=A0 > Now consider a situation where Renew lease feature is supported but=C2=A0= phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In = this case PhoenixConnection will keep adding the iterators for every scan i= nto the=C2=A0scannerQueue for renewal based on the feature supported check = but the renewal task is not running because=C2=A0phoenix.scanner.lease.rene= w.enabled is set to false, so the=C2=A0scannerQueue will keep growing as lo= ng as the PhoenixConnection is alive and multiple scans requests are coming= on this connection. > =C2=A0 > We have a use case that uses a single PhoenixConnection that is perpetual= and does billions of scans on this connection. In this case=C2=A0scannerQu= eue is growing to several GB's and ultimately leading to Consecutive Full G= C's/OOM > =C2=A0 > Add iterators for Lease renewal in=C2=A0PhoenixConnection > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > {code:java} > =C2=A0 > public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) = { > if (services.supportsFeature(Feature.RENEW_LEASE)) > {=20 > checkNotNull(itr); scannerQueue.add(new WeakReference(itr));=20 > } > } > {code} > =C2=A0 > Starting the=C2=A0RenewLeaseTask > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D > checks if=C2=A0Feature.RENEW_LEASE is supported and if phoenix.scanner.le= ase.renew.enabled is true and starts the=C2=A0RenewLeaseTask > {code:java} > =C2=A0 > ConnectionQueryServicesImpl { > .... > this.renewLeaseEnabled =3D config.getBoolean(RENEW_LEASE_ENABLED, DEFAULT= _RENEW_LEASE_ENABLED); > ..... > @Override > public boolean isRenewingLeasesEnabled(){=20 > return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) &&= renewLeaseEnabled;=20 > } > private void scheduleRenewLeaseTasks() { > if (isRenewingLeasesEnabled()) { > renewLeaseExecutor =3D > Executors.newScheduledThreadPool(renewLeasePoolSize, renewLeaseThreadF= actory); > for (LinkedBlockingQueue> q : connect= ionQueues) {=20 > renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, ren= ewLeaseTaskFrequency, TimeUnit.MILLISECONDS);=20 > } > } > } > ....... > } > {code} > =C2=A0 > To solve this We must add both checks in PhoenixConnection if the feature= is supported and if the config is enabled before adding the iterators to s= cannerQueue > ConnectionQueryServices.Feature.RENEW_LEASE is true=C2=A0 &&=C2=A0 phoeni= x.scanner.lease.renew.enabled is true=C2=A0 > instead of just checking if the feature=C2=A0ConnectionQueryServices.Feat= ure.RENEW_LEASE is supported > =C2=A0 > =C2=A0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)