Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CEFA811904 for ; Tue, 8 Jul 2014 11:34:16 +0000 (UTC) Received: (qmail 85543 invoked by uid 500); 8 Jul 2014 11:34:15 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 85495 invoked by uid 500); 8 Jul 2014 11:34:14 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 85485 invoked by uid 99); 8 Jul 2014 11:34:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jul 2014 11:34:14 +0000 X-ASF-Spam-Status: No, hits=2.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of bbowman410@gmail.com designates 209.85.216.53 as permitted sender) Received: from [209.85.216.53] (HELO mail-qa0-f53.google.com) (209.85.216.53) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jul 2014 11:34:10 +0000 Received: by mail-qa0-f53.google.com with SMTP id j15so4713769qaq.40 for ; Tue, 08 Jul 2014 04:33:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=wlSW6smYCUw2+U2L5fFg1XNAUd6NTt1Z8vwlc3NzShM=; b=kkPDqxwVdhaOTWiFrO5iBqUCvlWbVJoBQnhwMa0TEJTxoGy3Yxf1K4RGYqg8giMdws 66sWOfmIX5IFgRAloLu1xkHVbu0uJNqxv98YFccixzLJ+M6kl24aEFEYY5SuwzGX3HMp pfgrTQ6HP3dXq0zwXbze3fRupN4kDQ4+loCfKvg3O4aAV0d5wkY7vRTnbGWmSdu8Lp3u npDGLnmU6LM+c1E903+L+Q/mW14zJTYLPp1YPLxRrdbrzlAbALBZ1W9zosXAu2N1n1xa d2zDT61hhGX/LpIMBeq1iqPia0gpTz6tZ6tbgouxGB81DxL2soxV1M7BJDV2h5lmJ7wQ KPBQ== MIME-Version: 1.0 X-Received: by 10.140.27.215 with SMTP id 81mr5895431qgx.18.1404819229472; Tue, 08 Jul 2014 04:33:49 -0700 (PDT) Received: by 10.229.158.135 with HTTP; Tue, 8 Jul 2014 04:33:49 -0700 (PDT) In-Reply-To: References: Date: Tue, 8 Jul 2014 07:33:49 -0400 Message-ID: Subject: Re: Possible memory leak with 0.13 and JDBC From: Benjamin Bowman To: "jonas.partner" Cc: =?UTF-8?B?TmF2aXPrpZjsirnsmrA=?= , user@hive.apache.org Content-Type: multipart/alternative; boundary=001a11c14ae0257da904fdacf78f X-Virus-Checked: Checked by ClamAV on apache.org --001a11c14ae0257da904fdacf78f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hello again, Some progress has been made on this issue. From initial testing this patch has fixed my problem. I had my cluster running all night and the memory usage is floating around 700 MB. Before it would be > 1GB and climbing. https://issues.apache.org/jira/browse/HIVE-7353 -Benjamin On Tue, Jul 8, 2014 at 5:51 AM, jonas.partner wrote: > Hi Navis, > > after a run to the point where we are seeing exceptions we see the below > > num #instances #bytes class name > ---------------------------------------------- > 1: 502562 16081984 java.util.HashMap$Entry > 2: 459139 14692448 java.util.Hashtable$Entry > 3: 448636 10804840 [Ljava.lang.String; > 4: 57549 8853936 > 5: 57549 7377040 > 6: 4499 6273896 > 7: 109643 5151728 [C > 8: 923 4626864 [Ljava.util.Hashtable$Entry; > 9: 7783 3832416 [Ljava.util.HashMap$Entry; > 10: 4499 3214832 > 11: 3563 3208160 > 12: 41943 2612856 [Ljava.lang.Object; > 13: 107940 2590560 java.lang.String > 14: 14584 1934088 [S > 15: 76049 1825176 java.util.ArrayList > 16: 16139 1592936 [B > 17: 2303 1041680 > 18: 40389 969336 javax.ws.rs.core.MediaType > 19: 33764 810336 > com.sun.jersey.core.spi.component.ProviderFactory$SingletonComponentProv= ider > 20: 33764 810336 > com.sun.jersey.core.spi.component.ComponentInjector > 21: 25690 616560 > com.sun.jersey.core.spi.factory.MessageBodyFactory$MessageBodyWriterPair > 22: 4877 613240 java.lang.Class > 23: 11504 552192 java.util.HashMap > 24: 33764 540224 > com.sun.jersey.core.spi.component.ComponentDestructor > 25: 14680 469760 > com.sun.jersey.core.util.KeyComparatorHashMap$Entry > 26: 7927 436512 [[I > 27: 2936 234880 > [Lcom.sun.jersey.core.util.KeyComparatorHashMap$Entry; > 28: 6886 220352 > java.util.concurrent.ConcurrentHashMap$HashEntry > 29: 8833 211992 java.util.LinkedList$Node > 30: 13212 211392 > com.sun.jersey.core.impl.provider.xml.ThreadLocalSingletonContextProvide= r$2 > 31: 644 204520 [I > 32: 8074 193776 > com.sun.jersey.core.spi.factory.InjectableProviderFactory$MetaInjectable= Provider > 33: 361 190608 > 34: 4178 167120 java.util.LinkedHashMap$Entry > 35: 8919 142704 java.lang.Object > 36: 2936 140928 > com.sun.jersey.core.util.KeyComparatorHashMap > 37: 2463 137928 java.util.LinkedHashMap > 38: 2959 118360 java.lang.ref.Finalizer > 39: 1345 107600 java.lang.reflect.Method > 40: 2341 93640 java.util.WeakHashMap$Entry > 41: 3670 88080 > com.sun.jersey.api.client.Client$ContextInjectableProvider > 42: 734 82208 > org.apache.hadoop.hive.ql.hooks.ATSHook$1 > 43: 2470 79040 > java.util.concurrent.locks.ReentrantLock$NonfairSync > 44: 747 77688 java.lang.Thread > 45: 2936 70464 > com.sun.jersey.core.impl.provider.xml.ThreadLocalSingletonContextProvide= r$1 > 46: 2202 70464 > org.codehaus.jackson.jaxrs.MapperConfigurator > 47: 752 68672 [Ljava.util.WeakHashMap$Entry; > 48: 218 65552 > [Ljava.util.concurrent.ConcurrentHashMap$HashEntry; > 49: 1569 64656 [J > 50: 1613 64520 java.util.TreeMap$Entry > > where as at the start it looked like > > num #instances #bytes class name > ---------------------------------------------- > 1: 42631 6000544 > 2: 42631 5467456 > 3: 3107 4158440 > 4: 3107 2280128 > 5: 2460 2099296 > 6: 14455 1592728 [C > 7: 9631 951424 [B > 8: 3357 411624 java.lang.Class > 9: 5254 351232 [S > 10: 14281 342744 java.lang.String > 11: 5456 314848 [[I > 12: 654 309600 > 13: 5692 182144 java.util.HashMap$Entry > 14: 4826 154432 java.util.Hashtable$Entry > 15: 4160 133120 > java.util.concurrent.ConcurrentHashMap$HashEntry > 16: 233 123024 > 17: 3654 95232 [Ljava.lang.String; > 18: 1044 88912 [Ljava.lang.Object; > 19: 357 72512 [Ljava.util.HashMap$Entry; > 20: 4037 64592 java.lang.Object > 21: 1608 64320 java.util.TreeMap$Entry > 22: 1566 62640 > com.google.common.collect.MapMakerInternalMap$WeakEntry > 23: 880 56320 java.net.URL > 24: 167 51048 [I > 25: 559 44720 java.lang.reflect.Method > 26: 86 40080 [Ljava.util.Hashtable$Entry; > 27: 163 36512 > [Ljava.util.concurrent.ConcurrentHashMap$HashEntry; > 28: 779 31160 java.util.LinkedHashMap$Entry > 29: 370 29664 [Ljava.util.WeakHashMap$Entry; > 30: 457 29248 > org.apache.hadoop.hive.conf.HiveConf$ConfVars > 31: 726 29040 java.lang.ref.Finalizer > 32: 335 26800 java.util.jar.JarFile$JarFileEntry > 33: 1566 25056 > com.google.common.collect.MapMakerInternalMap$StrongValueReference > 34: 351 22464 java.util.jar.JarFile > 35: 370 20720 java.util.WeakHashMap > 36: 438 17520 java.lang.ref.SoftReference > 37: 360 17280 sun.nio.cs.UTF_8$Encoder > 38: 358 17184 sun.misc.URLClassPath$JarLoader > 39: 337 16176 java.util.zip.Inflater > 40: 223 16056 java.lang.reflect.Constructor > 41: 328 15744 java.util.HashMap > 42: 537 12888 java.util.LinkedList$Node > 43: 539 12520 [Ljava.lang.Class; > 44: 384 12288 java.lang.ref.ReferenceQueue > 45: 357 11424 java.util.zip.ZipCoder > 46: 284 9088 java.util.LinkedList > 47: 357 8568 java.util.ArrayDeque > 48: 337 8088 java.util.zip.ZStreamRef > 49: 306 7344 java.lang.Long > 50: 34 7176 [Z > > > > On 8 July 2014 at 08:40:20, Navis=EB=A5=98=EC=8A=B9=EC=9A=B0 (navis.ryu@n= exr.com > ) wrote: > > Could you try "jmap -histo:live " and check hive objects which seeme= d > too many? > > Thanks, > Navis > > > 2014-07-07 22:22 GMT+09:00 jonas.partner : > >> Hi Benjamin, >> Unfortunately this was a really critical issue for us and I didn=E2=80= =99t think >> we would find a fix in time so we switched to generating a hive scripts >> programmatically then running that via an Oozie action which uses the Hi= ve >> CLI. This seems to create a stable solution although is a lot less >> convenient than JDBC for our use case. >> >> I hope to find some more time to look at this later in the week since >> JDBC would simplify the solution. I would be very interested to hear if >> you make any progress. >> >> Regards >> >> Jonas >> >> On 7 July 2014 at 14:14:46, Benjamin Bowman (bbowman410@gmail.com >> ) wrote: >> >> I believe I am having the same issue. Hive 0.13 and Hadoop 2.4. We >> had to increase the Hive heap to 4 GB which allows Hive to function for >> about 2-3 days. After that point it has consumed the entire heap and >> becomes unresponsive and/or throws OOM exceptions. We are using Beelin= e >> and HiveServer 2 and connect via JDBC to the database tens of thousands = of >> times a day. >> >> I have been working with a developer at Hortonworks to find a solution >> but we have not come up with anything yet. Have you made any progress o= n >> this issue? >> >> Thanks, >> Benjamin >> >> >> On Thu, Jul 3, 2014 at 4:17 PM, jonas.partner < >> jonas.partner@opencredo.com> wrote: >> >>> Hi Edward, >>> >>> Thanks for the response. Sorry I posted the wrong version. I also >>> added close on the two result sets to the code taken from the wiki as >>> below but still the same problem. >>> >>> Will try to run it through your kit at the weekend. For the moment I >>> switched to running the statements as a script through the hive client = (not >>> beeline) which seems stable even with hundreds of repetitions. >>> >>> Regards >>> >>> Jonas >>> >>> public static void run() throws SQLException { >>> try { >>> Class.forName(driverName); >>> } catch (ClassNotFoundException e) { >>> // TODO Auto-generated catch block >>> e.printStackTrace(); >>> System.exit(1); >>> } >>> //replace "hive" here with the name of the user the queries >>> should run as >>> Connection con =3D >>> DriverManager.getConnection("jdbc:hive2://localhost:10000/default", "hi= ve", >>> ""); >>> Statement stmt =3D con.createStatement(); >>> String tableName =3D "testHiveDriverTable"; >>> stmt.execute("drop table if exists " + tableName); >>> stmt.execute("create external table " + tableName + " (key >>> int, value string)"); >>> // show tables >>> String sql =3D "show tables '" + tableName + "'"; >>> System.out.println("Running: " + sql); >>> ResultSet res =3D stmt.executeQuery(sql); >>> if (res.next()) { >>> System.out.println(res.getString(1)); >>> } >>> res.close(); >>> // describe table >>> sql =3D "describe " + tableName; >>> System.out.println("Running: " + sql); >>> res =3D stmt.executeQuery(sql); >>> >>> while (res.next()) { >>> System.out.println(res.getString(1) + "\t" + >>> res.getString(2)); >>> } >>> res.close(); >>> stmt.close(); >>> con.close(); >>> } >>> >>> >>> >>> On 3 July 2014 at 21:05:25, Edward Capriolo (edlinuxguru@gmail.com >>> ) wrote: >>> >>> Not saying there is not a leak elswhere but >>> statement and resultset objects both have .close() >>> >>> Java 7 now allows you to autoclose >>> try ( Connection conn ...; Statement st =3D conn.createStatement() ){ >>> something >>> } >>> >>> >>> On Thu, Jul 3, 2014 at 6:35 AM, jonas.partner < >>> jonas.partner@opencredo.com> wrote: >>> >>>> We have been struggling to get a reliable system working where we >>>> interact with Hive over JDBC a lot. The pattern we see is that everyt= hing >>>> starts ok but the memory used by the Hive server process grows over ti= me >>>> and after some hundreds of operations we start to see exceptions. >>>> >>>> To ensure there was nothing stupid in our code causing this I took th= e >>>> example code from the wiki page for Hive 2 clients and put that in a l= oop. >>>> For us after about 80 runs we would see exceptions as below. >>>> >>>> 2014-04-21 07:31:02,251 ERROR [pool-5-thread-5]: >>>> server.TThreadPoolServer (TThreadPoolServer.java:run(215)) - Error occ= urred >>>> during processing of message. >>>> java.lang.RuntimeException: >>>> org.apache.thrift.transport.TTransportException >>>> at >>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(= TSaslServerTransport.java:219) >>>> at >>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPo= olServer.java:189) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j= ava:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.= java:615) >>>> at java.lang.Thread.run(Thread.java:744) >>>> Caused by: org.apache.thrift.transport.TTransportException >>>> at >>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport= .java:132) >>>> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >>>> at >>>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTra= nsport.java:178) >>>> at >>>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessag= e(TSaslServerTransport.java:125) >>>> at >>>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:25= 3) >>>> at >>>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTrans= port.java:41) >>>> at >>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(= TSaslServerTransport.java:216) >>>> ... 4 more >>>> >>>> This is also sometimes accompanied by out of memory exceptions. >>>> >>>> >>>> The code on the wiki did not close statements and adding that in >>>> changes the behaviour instead of exceptions things just lock up after = a >>>> while and there is high CPU usage. >>>> >>>> This looks similar to HIVE-5296 >>>> but that was fixed >>>> in 0.12 so should not be an issue in 0.13 I assume. Issues fixed in 0= .13.1 >>>> don=E2=80=99t seem to relate to this either. The only way to get Hive= back up and >>>> running is to restart. >>>> >>>> Before raising a JIRA I wanted to make sure I wasn=E2=80=99t missing >>>> something so any suggestions would be greatly appreciated. >>>> >>>> Full code as below. >>>> >>>> import java.sql.*; >>>> >>>> >>>> public class HiveOutOfMem { >>>> >>>> private static String driverName =3D >>>> "org.apache.hive.jdbc.HiveDriver"; >>>> >>>> >>>> public static void main(String[] args) throws SQLException{ >>>> for(int i =3D0; i < 100000; i++){ >>>> System.out.println("Run number " + i); >>>> run(); >>>> } >>>> } >>>> >>>> /** >>>> * @param >>>> * @throws SQLException >>>> */ >>>> public static void run() throws SQLException { >>>> try { >>>> Class.forName(driverName); >>>> } catch (ClassNotFoundException e) { >>>> // TODO Auto-generated catch block >>>> e.printStackTrace(); >>>> System.exit(1); >>>> } >>>> //replace "hive" here with the name of the user the querie= s >>>> should run as >>>> Connection con =3D >>>> DriverManager.getConnection("jdbc:hive2://localhost:10000/default", "h= ive", >>>> ""); >>>> Statement stmt =3D con.createStatement(); >>>> String tableName =3D "testHiveDriverTable"; >>>> stmt.execute("drop table if exists " + tableName); >>>> stmt.execute("create external table " + tableName + " (ke= y >>>> int, value string)"); >>>> // show tables >>>> String sql =3D "show tables '" + tableName + "'"; >>>> System.out.println("Running: " + sql); >>>> ResultSet res =3D stmt.executeQuery(sql); >>>> if (res.next()) { >>>> System.out.println(res.getString(1)); >>>> } >>>> >>>> // describe table >>>> sql =3D "describe " + tableName; >>>> System.out.println("Running: " + sql); >>>> res =3D stmt.executeQuery(sql); >>>> while (res.next()) { >>>> System.out.println(res.getString(1) + "\t" + >>>> res.getString(2)); >>>> } >>>> //stmt.close(); >>>> con.close(); >>>> } >>>> >>>> } >>>> >>> >>> >> > --001a11c14ae0257da904fdacf78f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hello again,

Some progress has been mad= e on this issue. =C2=A0From initial testing this patch has fixed my problem= . =C2=A0I had my cluster running all night and the memory usage is floating= around 700 MB. =C2=A0Before it would be > 1GB and climbing.

-Benjamin=C2=A0


<= div class=3D"gmail_quote"> On Tue, Jul 8, 2014 at 5:51 AM, jonas.partner <jonas.partner@ope= ncredo.com> wrote:
Hi Nav= is,

after a run to the point where w= e are seeing exceptions we see the below =C2=A0

num = =C2=A0 =C2=A0 #instances =C2=A0 =C2=A0 =C2=A0 =C2=A0 #bytes =C2=A0class nam= e
------------------------------------------= ----
=C2=A0 =C2=A01: =C2=A0 =C2=A0 =C2=A0 =C2=A0502562= =C2=A0 =C2=A0 =C2=A0 16081984 =C2=A0java.util.HashMap$Entry
=C2=A0 =C2=A02: =C2=A0 =C2=A0 =C2=A0 =C2=A0459139 =C2=A0 = =C2=A0 =C2=A0 14692448 =C2=A0java.util.Hashtable$Entry
=C2=A0 =C2=A03: =C2=A0 =C2=A0 =C2=A0 =C2=A0448636 =C2=A0 =C2=A0 = =C2=A0 10804840 =C2=A0[Ljava.lang.String;
=C2=A0 =C2=A04: =C2=A0 =C2=A0 =C2=A0 =C2=A0 57549= =C2=A0 =C2=A0 =C2=A0 =C2=A08853936 =C2=A0<constMethodKlass>
=C2=A0 =C2=A05: =C2=A0 =C2=A0 =C2=A0 =C2=A0 57549 = =C2=A0 =C2=A0 =C2=A0 =C2=A07377040 =C2=A0<methodKlass>
=C2=A0 =C2=A06: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A04499 =C2= =A0 =C2=A0 =C2=A0 =C2=A06273896 =C2=A0<constantPoolKlass>
=C2=A0 =C2=A07: =C2=A0 =C2=A0 =C2=A0 =C2=A0109643= =C2=A0 =C2=A0 =C2=A0 =C2=A05151728 =C2=A0[C
=C2=A0 =C2=A08: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 923 =C2=A0 =C2=A0 =C2= =A0 =C2=A04626864 =C2=A0[Ljava.util.Hashtable$Entry;
=C2=A0 =C2=A09: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A07783 =C2=A0 =C2= =A0 =C2=A0 =C2=A03832416 =C2=A0[Ljava.util.HashMap$Entry;
=C2=A0 10: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A04499= =C2=A0 =C2=A0 =C2=A0 =C2=A03214832 =C2=A0<instanceKlassKlass>
<= div style=3D"margin:0px">=C2=A0 11: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03563 = =C2=A0 =C2=A0 =C2=A0 =C2=A03208160 =C2=A0<constantPoolCacheKlass>
=C2=A0 12: =C2=A0 =C2=A0 =C2=A0 =C2=A0 41943 = =C2=A0 =C2=A0 =C2=A0 =C2=A02612856 =C2=A0[Ljava.lang.Object;
=C2=A0 13: =C2=A0 =C2=A0 =C2=A0 =C2=A0107940 =C2= =A0 =C2=A0 =C2=A0 =C2=A02590560 =C2=A0java.lang.String
=C2=A0 14: =C2=A0 =C2=A0 =C2=A0 =C2=A0 14584 =C2=A0 =C2=A0 =C2= =A0 =C2=A01934088 =C2=A0[S
=C2=A0 15: =C2=A0= =C2=A0 =C2=A0 =C2=A0 76049 =C2=A0 =C2=A0 =C2=A0 =C2=A01825176 =C2=A0java.u= til.ArrayList
=C2=A0 16: =C2=A0 =C2=A0 =C2=A0 =C2=A0 16139 =C2= =A0 =C2=A0 =C2=A0 =C2=A01592936 =C2=A0[B
=C2= =A0 17: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02303 =C2=A0 =C2=A0 =C2=A0 =C2=A01= 041680 =C2=A0<methodDataKlass>
=C2=A0 = 18: =C2=A0 =C2=A0 =C2=A0 =C2=A0 40389 =C2=A0 =C2=A0 =C2=A0 =C2=A0 969336 = =C2=A0javax.ws.rs.core.MediaType
=C2=A0 19: =C2=A0 =C2=A0 =C2=A0 =C2=A0 33764 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 810336 =C2=A0com.sun.jersey.core.spi.component.Pro= viderFactory$SingletonComponentProvider
=C2= =A0 20: =C2=A0 =C2=A0 =C2=A0 =C2=A0 33764 =C2=A0 =C2=A0 =C2=A0 =C2=A0 81033= 6 =C2=A0com.sun.jersey.core.spi.component.ComponentInjector
=C2=A0 21: =C2=A0 =C2=A0 =C2=A0 =C2=A0 25690 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 616560 =C2=A0com.sun.jersey.core.spi.factory.Messa= geBodyFactory$MessageBodyWriterPair
=C2=A0 2= 2: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A04877 =C2=A0 =C2=A0 =C2=A0 =C2=A0 61324= 0 =C2=A0java.lang.Class
=C2=A0 23: =C2=A0 =C2=A0 =C2=A0 =C2=A0 11504 =C2=A0 =C2=A0 =C2=A0 =C2=A0 55= 2192 =C2=A0java.util.HashMap
=C2=A0 24: =C2= =A0 =C2=A0 =C2=A0 =C2=A0 33764 =C2=A0 =C2=A0 =C2=A0 =C2=A0 540224 =C2=A0com= .sun.jersey.core.spi.component.ComponentDestructor
=C2=A0 25: =C2=A0 =C2=A0 =C2=A0 =C2=A0 14680 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 469760 =C2=A0com.sun.jersey.core.util.KeyComparatorHashMap$Entry
=C2=A0 26: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A07927= =C2=A0 =C2=A0 =C2=A0 =C2=A0 436512 =C2=A0[[I
=C2=A0 27: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02936 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 234880 =C2=A0[Lcom.sun.jersey.core.util.KeyComparatorHashMap$Entry;<= /div>
=C2=A0 28: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A06886 =C2=A0 =C2=A0 =C2=A0 =C2=A0 220352 =C2=A0java.util.concurrent.Concu= rrentHashMap$HashEntry
=C2=A0 29: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A08833= =C2=A0 =C2=A0 =C2=A0 =C2=A0 211992 =C2=A0java.util.LinkedList$Node
=C2=A0 30: =C2=A0 =C2=A0 =C2=A0 =C2=A0 13212 =C2=A0= =C2=A0 =C2=A0 =C2=A0 211392 =C2=A0com.sun.jersey.core.impl.provider.xml.Th= readLocalSingletonContextProvider$2
=C2=A0 31: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 644= =C2=A0 =C2=A0 =C2=A0 =C2=A0 204520 =C2=A0[I
=C2=A0 32: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A08074 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 193776 =C2=A0com.sun.jersey.core.spi.factory.InjectableProviderFactory$= MetaInjectableProvider
=C2=A0 33: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 361 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 190608 =C2=A0<objArrayKlassKlass>
= =C2=A0 34: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A04178 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 167120 =C2=A0java.util.LinkedHashMap$Entry
=C2=A0 35: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A08919 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 142704 =C2=A0java.lang.Object
=C2=A0 36: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02936= =C2=A0 =C2=A0 =C2=A0 =C2=A0 140928 =C2=A0com.sun.jersey.core.util.KeyCompa= ratorHashMap
=C2=A0 37: =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A02463 =C2=A0 =C2=A0 =C2=A0 =C2=A0 137928 =C2=A0java.util.Linke= dHashMap
=C2=A0 38: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02959 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 118360 =C2=A0java.lang.ref.Finalizer
=C2= =A0 39: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01345 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = 107600 =C2=A0java.lang.reflect.Method
=C2=A0= 40: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02341 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A093640 =C2=A0java.util.WeakHashMap$Entry
=C2=A0 41: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03670= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A088080 =C2=A0com.sun.jersey.api.client.Cl= ient$ContextInjectableProvider
=C2=A0 42: = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 734 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A082= 208 =C2=A0org.apache.hadoop.hive.ql.hooks.ATSHook$1
=C2=A0 43: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02470= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A079040 =C2=A0java.util.concurrent.locks.R= eentrantLock$NonfairSync
=C2=A0 44: =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 747 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A077688 =C2= =A0java.lang.Thread
=C2=A0 45: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02936 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A070464 =C2=A0com.sun.jersey.core.impl.provider.xml.ThreadLocalSing= letonContextProvider$1
=C2=A0 46: =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A02202 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A070464 =C2=A0= org.codehaus.jackson.jaxrs.MapperConfigurator
=C2=A0 47: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 752= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A068672 =C2=A0[Ljava.util.WeakHashMap$Entr= y;
=C2=A0 48: =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 218 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A065552 =C2=A0[Ljava.util.concur= rent.ConcurrentHashMap$HashEntry;
=C2=A0 49: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01569 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A064656 =C2=A0[J
=C2=A0 50: =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A01613 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A064520 =C2= =A0java.util.TreeMap$Entry

where as at the start it looked like

=C2=A0num =C2=A0 =C2=A0 #instances =C2=A0 =C2=A0 =C2=A0 =C2=A0= #bytes =C2=A0class name
-------------------= ---------------------------
=C2=A0 =C2=A01: =C2=A0 =C2=A0 =C2=A0 =C2=A0 42631 =C2=A0 =C2=A0 =C2=A0 =C2= =A06000544 =C2=A0<constMethodKlass>
= =C2=A0 =C2=A02: =C2=A0 =C2=A0 =C2=A0 =C2=A0 42631 =C2=A0 =C2=A0 =C2=A0 =C2= =A05467456 =C2=A0<methodKlass>
=C2=A0 = =C2=A03: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03107 =C2=A0 =C2=A0 =C2=A0 =C2=A0= 4158440 =C2=A0<constantPoolKlass>
=C2=A0 =C2=A04: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A03107 =C2=A0 =C2=A0 =C2=A0 =C2=A02280128 =C2=A0<instanceKlassKlass>=
=C2=A0 =C2=A05: =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A02460 =C2=A0 =C2=A0 =C2=A0 =C2=A02099296 =C2=A0<constantPoolCacheK= lass>
=C2=A0 =C2=A06: =C2=A0 =C2=A0 =C2= =A0 =C2=A0 14455 =C2=A0 =C2=A0 =C2=A0 =C2=A01592728 =C2=A0[C
=C2=A0 =C2=A07: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A09631 =C2=A0 =C2=A0 =C2=A0 =C2=A0 951424 =C2=A0[B
=C2=A0 =C2=A08: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03357 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 411624 =C2=A0java.lang.Class
=C2=A0 =C2=A09: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A05254 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 351232 =C2=A0[S
=C2=A0 10: =C2=A0 =C2=A0 =C2=A0 =C2=A0 14281 =C2=A0 =C2=A0 =C2=A0 =C2=A0 34= 2744 =C2=A0java.lang.String
=C2=A0 11: =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A05456 =C2=A0 =C2=A0 =C2=A0 =C2=A0 314848 =C2= =A0[[I
=C2=A0 12: =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 654 =C2=A0 =C2=A0 =C2=A0 =C2=A0 309600 =C2=A0<methodDataKlass= >
=C2=A0 13: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A05692 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 182144 =C2=A0java.util.HashMap$Entry
=C2= =A0 14: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A04826 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = 154432 =C2=A0java.util.Hashtable$Entry
=C2= =A0 15: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A04160 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = 133120 =C2=A0java.util.concurrent.ConcurrentHashMap$HashEntry
=C2=A0 16: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 233= =C2=A0 =C2=A0 =C2=A0 =C2=A0 123024 =C2=A0<objArrayKlassKlass>
<= div style=3D"margin:0px">=C2=A0 17: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03654 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A095232 =C2=A0[Ljava.lang.String;
=C2=A0 18: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01044 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A088912 =C2=A0[Ljava.lang.Object;
=C2=A0 19: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 357= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A072512 =C2=A0[Ljava.util.HashMap$Entry;
=C2=A0 20: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= 4037 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A064592 =C2=A0java.lang.Object
=C2=A0 21: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01608 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A064320 =C2=A0java.util.TreeMap$Entry
=C2=A0 22: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01566= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A062640 =C2=A0com.google.common.collect.Ma= pMakerInternalMap$WeakEntry
=C2=A0 23: =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 880 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A056320= =C2=A0java.net.URL
=C2=A0 24: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 167 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A051048 =C2=A0[I
=C2=A0 25: =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 559 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A044720 =C2= =A0java.lang.reflect.Method
=C2=A0 26: =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A086 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= 40080 =C2=A0[Ljava.util.Hashtable$Entry;
=C2=A0 27: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 163= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A036512 =C2=A0[Ljava.util.concurrent.Concu= rrentHashMap$HashEntry;
=C2=A0 28: =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 779 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A031160 =C2= =A0java.util.LinkedHashMap$Entry
=C2=A0 29: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 370 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A029664 =C2=A0[Ljava.util.WeakHashMap$Entry;
=C2=A0 30: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 457 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A029248 =C2=A0org.apache.hadoop.hive.conf.HiveConf$ConfVa= rs
=C2=A0 31: =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 726 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A029040 =C2=A0java.lang.ref.Fina= lizer
=C2=A0 32: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 335= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A026800 =C2=A0java.util.jar.JarFile$JarFil= eEntry
=C2=A0 33: =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A01566 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A025056 =C2=A0com.google.com= mon.collect.MapMakerInternalMap$StrongValueReference
=C2=A0 34: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 351= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A022464 =C2=A0java.util.jar.JarFile
<= div style=3D"margin:0px">=C2=A0 35: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 370 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A020720 =C2=A0java.util.WeakHashMap
=C2=A0 36: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 438 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A017520 =C2=A0java.lang.ref.SoftReference
=C2=A0 37: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 360= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A017280 =C2=A0sun.nio.cs.UTF_8$Encoder
=C2=A0 38: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 3= 58 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A017184 =C2=A0sun.misc.URLClassPath$JarL= oader
=C2=A0 39: =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 337 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A016176 =C2=A0java.util.zip.Inf= later
=C2=A0 40: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 223= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A016056 =C2=A0java.lang.reflect.Constructo= r
=C2=A0 41: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 328 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A015744 =C2=A0java.util.HashMap
=C2=A0 42: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 5= 37 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A012888 =C2=A0java.util.LinkedList$Node<= /div>
=C2=A0 43: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 539= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A012520 =C2=A0[Ljava.lang.Class;
=C2=A0 44: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 384 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A012288 =C2=A0java.lang.ref.ReferenceQueue
=C2=A0 45: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 3= 57 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A011424 =C2=A0java.util.zip.ZipCoder
=C2=A0 46: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 284= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 9088 =C2=A0java.util.LinkedList
=C2=A0 47: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 357 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 8568 =C2=A0java.util.ArrayDeque
=C2=A0 48: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 337 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 8088 =C2=A0java.util.zip.ZStreamRef
=C2=A0 49: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 306= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 7344 =C2=A0java.lang.Long
=C2=A0 50: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A034 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 7176 =C2=A0[Z



On 8 July 2014 at 08:40:20, Navis=EB=A5=98=EC=8A=B9=EC=9A=B0 (navis.ryu@nexr.com) wrote:

Could you try "jmap -histo:live <pid>" and check hive objects which seemed too many?

Thanks,=C2=A0
Navis


2014-07-07 22:22 GMT+09:00 jonas.partner <jonas.partner@opencredo.com>:
Hi Benjamin,
Unfortunately this was a really critical issue for us and I didn=E2=80=99t think we would find a fix in time so we switched =C2=A0to generating a hive scripts programmatically then running that via an Oozie action which uses the Hive CLI. =C2=A0This seems to create a stable solution although is a lot less convenient than JDBC for our use case.

I hope to find some more time to look at this later in the week since JDBC would simplify the solution. =C2=A0I would be very interested to hear if you make any progress.

Regards

Jonas

On 7 July 2014 at 14:14:46, Benjamin Bowman (bbowman= 410@gmail.com) wrote:

I believe I am having the same issue. =C2=A0Hive 0.13 and Hadoop 2.4. =C2=A0We had to increase the Hive heap to 4 GB which allows Hive to function for about 2-3 days. =C2=A0After that point it has consumed the entire heap and becomes unresponsive and/or throws OOM exceptions. =C2=A0We are using =C2=A0Beeline and HiveServer 2 and connect via JDBC to the database tens of thousands of times a day. =C2=A0

I have been working with a developer at Hortonworks to find a solution but we have not come up with anything yet. =C2=A0Have you made any progress on this issue?

Thanks,
Benjamin
On Thu, Jul 3, 2014 at 4:17 PM, jonas.partner <jonas.partner@opencredo.com> wrote:
Hi Edward,

Thanks for the response. =C2=A0Sorry I posted the wrong version. I also added close =C2=A0on the two result sets to the code taken from the wiki as below but still the same problem.

Will try to run it through your kit at the weekend. =C2=A0For the moment I switched to running the statements as a script through the hive client (not beeline) which seems stable even with hundreds of repetitions.

Regards

Jonas

=C2=A0public static void run() throws SQLException {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 try {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Class.forName(driverName);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 } catch (ClassNotFoundException e) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 // TODO Auto-generated catch block
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 e.printStackTrace();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 System.exit(1);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 //replace "hive" here with the name of the user the queries shoul= d run as
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Connection con =3D DriverManager.getConnection("jdbc:hive2://localhost:10000/default"= ;, "hive", "");
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Statement stmt =3D con.createStatement();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 String tableName =3D "testHiveDriverTable";
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stmt.execute("drop table if exists " + tableName);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stmt.execute("create external table =C2=A0" + tableName + " = (key int, value string)");
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 // show tables
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 String sql =3D "show tables '" + tableName + "'"= ;;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 System.out.println("Running: " + sql);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ResultSet res =3D stmt.executeQuery(sql);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (res.next()) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 System.out.println(res.getString(1));
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 res.close();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 // describe table
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sql =3D "describe " + tableName;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 System.out.println("Running: " + sql);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 res =3D stmt.executeQuery(sql);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 while (res.next()) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 System.out.println(res.getString(1) + "\t" + res.getString(2));
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 res.close();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stmt.close();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 con.close();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }

=C2=A0=C2=A0

On 3 July 2014 at 21:05:25, Edward Capriolo (edli= nuxguru@gmail.com) wrote:

Not saying there is not a leak elswhere but
statement and resultset objects both have .close()

Java 7 now allows you to autoclose
try (=C2=A0 Connection conn ...; Statement st =3D conn.createStatement() ){
something
}
On Thu, Jul 3, 2014 at 6:35 AM, jonas.partner <jonas.partner@opencredo.com> wrote:
We have been struggling to get a reliable system working where we interact with Hive over JDBC a lot. =C2=A0The pattern we see is that everything starts ok but the memory used by the Hive server process grows over time and after some hundreds of operations we start to see exceptions. =C2=A0

To ensure there was nothing stupid in our code causing this I took the example code from the wiki page for Hive 2 clients and put that in a loop. =C2=A0For us after about 80 runs we would see exceptions as below. =C2=A0=C2=A0

2014-04-21 07:31:02,251 ERROR [pool-5-thread-5]: server.TThreadPoolServer (TThreadPoolServer.java:run(215)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSasl= ServerTransport.java:219)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolSer= ver.java:189)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1= 145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:= 615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java= :132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTranspor= t.java:178)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSa= slServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.= java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSasl= ServerTransport.java:216)
... 4 more

This is also sometimes accompanied by out of memory exceptions.


The code on the wiki did not close statements and adding that in changes the behaviour instead of exceptions things just lock up after a while and there is high CPU usage.

This looks similar to=C2=A0HIVE-5296=C2=A0but that wa= s fixed in 0.12 so should not be an issue in 0.13 I assume. =C2=A0Issues fixed in 0.13.1 don=E2=80=99t seem to relate to this either. =C2=A0The only way to get Hive back up and running is to restart.
=C2=A0 =C2=A0 =C2=A0
Before raising a JIRA I wanted to make sure I wasn=E2=80=99t missing something so any suggestions would be greatly appreciated. =C2=A0

Full code as below.

import java.sql.*;


public class HiveOutOfMem {

=C2=A0 =C2=A0 =C2=A0 =C2=A0 private static String driverName =3D "org.apache.hive.jdbc.HiveDriver";


=C2=A0 =C2=A0 =C2=A0 =C2=A0 public static void main(String[] args) throws SQLException{
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 for(int i =3D0; i < 100000; i++){
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 System.out.println("Run number " + i);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 run();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }

=C2=A0 =C2=A0 =C2=A0 =C2=A0 /**
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* @param
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* @throws SQLException
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*/
=C2=A0 =C2=A0 =C2=A0 =C2=A0 public static void run() throws SQLException {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 try {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Class.forName(driverName);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 } catch (ClassNotFoundException e) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 // TODO Auto-generated catch block
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 e.printStackTrace();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 System.exit(1);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 //replace "hive" here with the name of the user the queries should run as
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Connection con =3D DriverManager.getConnection("jdbc:hive2://localhost:10000/default"= ;, "hive", "");
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Statement stmt =3D con.createStatement();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 String tableName =3D "testHiveDriverTable";
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stmt.execute("drop table if exists " + tableName);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stmt.execute("create external table =C2=A0" + tableName + " (key int, value string)");
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 // show tables
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 String sql =3D "show tables '" + tableName + &q= uot;'";
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 System.out.println("Running: " + sql);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ResultSet res =3D stmt.executeQuery(sql);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (res.next()) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 System.out.println(res.getString(1));
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 // describe table
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sql =3D "describe " + tableName;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 System.out.println("Running: " + sql);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 res =3D stmt.executeQuery(sql);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 while (res.next()) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 System.out.println(res.getString(1) + "\t" + res.getString(2));
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 //stmt.close();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 con.close();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }

}




--001a11c14ae0257da904fdacf78f--