Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 15F5510A86 for ; Tue, 6 Aug 2013 23:40:29 +0000 (UTC) Received: (qmail 62626 invoked by uid 500); 6 Aug 2013 23:40:26 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 62535 invoked by uid 500); 6 Aug 2013 23:40:26 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 62220 invoked by uid 500); 6 Aug 2013 23:40:26 -0000 Delivered-To: apmail-hadoop-hive-user@hadoop.apache.org Received: (qmail 62212 invoked by uid 99); 6 Aug 2013 23:40:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Aug 2013 23:40:26 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of echu@rocketfuelinc.com designates 74.125.245.80 as permitted sender) Received: from [74.125.245.80] (HELO na3sys010aog106.obsmtp.com) (74.125.245.80) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Aug 2013 23:40:21 +0000 Received: from mail-ob0-f181.google.com ([209.85.214.181]) (using TLSv1) by na3sys010aob106.postini.com ([74.125.244.12]) with SMTP ID DSNKUgGJUKKv/w0aQTC1BugQk5kz45muiCiZ@postini.com; Tue, 06 Aug 2013 16:40:01 PDT Received: by mail-ob0-f181.google.com with SMTP id dn14so2308091obc.40 for ; Tue, 06 Aug 2013 16:40:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rocketfuel.com; s=google; h=mime-version:from:date:message-id:subject:to:content-type; bh=2zpL/dVV4cdWZwRpfJSbJlxcc53jj+pTc5/SXPwZXIc=; b=SWhZlgXrKRp2hven0ujYcv2hLLj7NjGJhBRNosuiiXh+fbq9LSIlLrMge0h+d7tbEA JQMBUovl5c7vkiZgNnKzpLcI3qpAtgTaubC53qvLLj+2Hhvmbl9Q3eksiVGvFpdPHH6b BrPmB1x40xnGThSjMmCZXsTGw0OMBzx7JbFN4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-gm-message-state:mime-version:from:date:message-id:subject:to :content-type; bh=2zpL/dVV4cdWZwRpfJSbJlxcc53jj+pTc5/SXPwZXIc=; b=R/cJPA+GJuYdasjKits2VS4qst2vOcBMpJwOxOSqLBihVI4oGTcMZExTZzel++ZWJg hZl4SzkOEh257lnjUDm4IVRU/dPG9+f+AOhRfx4tW0AFaE3v0AsF7LbngKs16rB10Y3v Xzv1gdmWwm+SeHKmlSeqXJ5LJ+hqSixWaGdYu4lcL2XfT360Kzb7doLgEi74zCiyVkKK +9A0JJfIRkxTLmPbKPuoQDi23uDEWPO/jt1b/q4yw1mLdJS0xqN1ZWMztqvzTpllUERe ed7TXpWvhhFLSlrXCpEFn0ZiPDPwbYk2ThITNO3m/FWI4tGcjMC3bjBOSdY+PuRDZHwj AQWw== X-Gm-Message-State: ALoCoQlt2m24Q9bv0QtS4FsCo0xC6kQa6X6UFJTMe5TrNMGre7pKpANOT8nzXLMpbSRQ2whks7pCqVLEM4E78gRpxEsv79nDXtSqIyLOjUmVjAcUrx9Mib9Z4lTWL8W7TtSaZB71oR+CeBPvY5fmctT6neY/5Esme6aSSl8EN065QFMGp7Z26Hg= X-Received: by 10.182.99.232 with SMTP id et8mr427499obb.61.1375832400436; Tue, 06 Aug 2013 16:40:00 -0700 (PDT) X-Received: by 10.182.99.232 with SMTP id et8mr427496obb.61.1375832400289; Tue, 06 Aug 2013 16:40:00 -0700 (PDT) MIME-Version: 1.0 Received: by 10.60.10.165 with HTTP; Tue, 6 Aug 2013 16:39:40 -0700 (PDT) From: Eric Chu Date: Tue, 6 Aug 2013 16:39:40 -0700 Message-ID: Subject: DataNucleus patches for Hive To: hive-user@hadoop.apache.org, dev@hive.apache.org Content-Type: multipart/alternative; boundary=089e0158b20a7d745c04e34ff1a2 X-Virus-Checked: Checked by ClamAV on apache.org --089e0158b20a7d745c04e34ff1a2 Content-Type: text/plain; charset=ISO-8859-1 Hi, I'm a bit confused about what DataNucleus patches we should get for *Hive 0.11 with JDK 6*. It'd be great if people working on that could shed some light on the subject. Thanks in advance! After installing Hive 0.11 and applying the patch for HIVE-4619 (or else MR queries will result in error), I will sometimes (but not always) see DataNucleus error log messages as below: 13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core" requires "org.eclipse.core.resources" but it cannot be resolved. 13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core" requires "org.eclipse.core.runtime" but it cannot be resolved. 13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.jdt.core" requires "org.eclipse.text" but it cannot be resolved. HIVE-2015 was opened for this issue, but the last comment there said it was fixed through HIVE-3632, whose purpose is to upgrade DataNucleus to support JDK 7. HIVE-3632 in turn was blocked by HIVE-4900 and broke HIVE-4942, both of which have patches available in 12. So I ended up applying HIVE-4900, HIVE-3632, and HIVE-4942, in that order. However, afterward, any query (MR, select *, show tables) would result in error. In the end, after I applied the patch for HIVE-4900, I haven't seen those error messages and queries are still running fine. Are the other patches not compatible with JDK 6? *Error messages for SHOW TABLES after applying HIVE-4900, HIVE-3632, and HIVE-4942:* FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient 13/08/06 12:31:01 ERROR exec.Task: FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1140) at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1125) at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:2265) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:337) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:144) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1361) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1146) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:948) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:261) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:218) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:415) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:758) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1139) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:51) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2284) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2295) at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1136) ... 18 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1137) ... 23 more Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception caught. NestedThrowables: java.lang.reflect.InvocationTargetException at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186) at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803) at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698) at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:262) at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:291) at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:224) at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:199) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130) at org.apache.hadoop.hive.metastore.RetryingRawStore.(RetryingRawStore.java:62) at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:413) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:401) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:439) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:325) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:285) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:53) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:58) at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4102) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:121) ... 28 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958) at java.security.AccessController.doPrivileged(Native Method) at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953) at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159) ... 47 more Caused by: java.lang.NoSuchFieldError: NUCLEUS_CONTEXT_LOADER at org.datanucleus.NucleusContext.(NucleusContext.java:105) at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.(JDOPersistenceManagerFactory.java:395) at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:294) at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195) ... 55 more FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask 13/08/06 12:31:01 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask --089e0158b20a7d745c04e34ff1a2 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi,

I'm a bit confused about what DataNucleus patches w= e should get for Hive 0.11 with JDK 6. It'd be great if people w= orking on that could shed some light on the subject. Thanks in advance!

After installing Hive 0.11 and applying the patch for HIVE-4619 (or else MR= queries will result in error), I will sometimes (but not always) see DataN= ucleus error log messages as below:

13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.j= dt.core" requires "org.eclipse.core.resources" but it cannot= be resolved.
13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle = "org.eclipse.jdt.core" requires "org.eclipse.core.runtime&qu= ot; but it cannot be resolved.
13/08/06 12:32:33 ERROR DataNucleus.Plugin: Bundle "org.eclipse.j= dt.core" requires "org.eclipse.text" but it cannot be resolv= ed.=A0

HIVE-2015 was opened for this issue, but th= e last comment there said it was fixed through HIVE-3632, whose purpose is = to upgrade DataNucleus to support JDK 7. HIVE-3632 in turn was blocked by H= IVE-4900 and broke HIVE-4942, both of which have patches available in 12. S= o I ended up applying HIVE-4900, HIVE-3632, and HIVE-4942, in that order. H= owever, afterward, any query (MR, select *, show tables) would result in er= ror.=A0

In the end, after I applied the patch for HIVE-4900, I = haven't seen those error messages and queries are still running fine. A= re the other patches not compatible with JDK 6?

Error messages for SHOW TABLES after applying HIVE-4= 900, HIVE-3632, and HIVE-4942:

FAILED: Er= ror in metadata: java.lang.RuntimeException: Unable to instantiate org.apac= he.hadoop.hive.metastore.HiveMetaStoreClient
13/08/06 12:31:01 ERROR exec.Task: FAILED: Error in metadata: java.lan= g.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.= HiveMetaStoreClient
org.apache.hadoop.hive.ql.metadata.HiveExcept= ion: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hi= ve.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive= .ql.metadata.Hive.getDatabase(Hive.java:1140)
at org.apache.hadoop.hive.ql.metadata.Hive.databa= seExists(Hive.java:1125)
at org.apache.hadoop.hive= .ql.exec.DDLTask.showTables(DDLTask.java:2265)
at org.apache.hadoop.hive.ql.exec.DDLTask.execut= e(DDLTask.java:337)
at org.apache.hadoop.hive= .ql.exec.Task.executeTask(Task.java:144)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequen= tial(TaskRunner.java:57)
at org.apache.hadoop.hive= .ql.Driver.launchTask(Driver.java:1361)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java= :1146)
at org.apache.hadoop.hive= .ql.Driver.run(Driver.java:948)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriv= er.java:261)
at org.apache.hadoop.hive= .cli.CliDriver.processCmd(CliDriver.java:218)
at org.apache.hadoop.hive.cli.CliDriver.processLi= ne(CliDriver.java:415)
at org.apache.hadoop.hive= .cli.CliDriver.run(CliDriver.java:758)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.j= ava:616)
at sun.reflect.NativeMeth= odAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMeth= odAccessorImpl.java:39)
at sun.reflect.Delegating= MethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
<= span style=3D"white-space:pre-wrap"> at java.lang.reflect.Method.inv= oke(Method.java:597)
at org.apache.hadoop.util= .RunJar.main(RunJar.java:208)
Caused by: java.lang.RuntimeExcepti= on: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreCli= ent
at org.apache.hadoop.hive= .metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1139)
<= span style=3D"white-space:pre-wrap"> at org.apache.hadoop.hive.metas= tore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:51)<= /div>
at org.apache.hadoop.hive= .metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61= )
at org.apache.hado= op.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2284)
at org.apache.hadoop.hive= .ql.metadata.Hive.getMSC(Hive.java:2295)
at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase= (Hive.java:1136)
... 18 more
Cau= sed by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorI= mpl.newInstance0(Native Method)
at sun.reflect.NativeCons= tructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.Delegatin= gConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java= :27)
at java.lang.reflect.Cons= tructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.ne= wInstance(MetaStoreUtils.java:1137)
... 23 more
Cau= sed by: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetExce= ption
at javax.jdo.JDOHelper.in= vokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186)
=
at javax.jdo.JDOHelper.ge= tPersistenceManagerFactory(JDOHelper.java:803)
at javax.jdo.JDOHelper.ge= tPersistenceManagerFactory(JDOHelper.java:698)
at org.apache.hadoop.hive.metastore.ObjectStore.= getPMF(ObjectStore.java:262)
at org.apache.hadoop.hive= .metastore.ObjectStore.getPersistenceManager(ObjectStore.java:291)
at org.apache.hadoop.hive.me= tastore.ObjectStore.initialize(ObjectStore.java:224)
at org.apache.hadoop.hive= .metastore.ObjectStore.setConf(ObjectStore.java:199)
at org.apache.hadoop.util.ReflectionUtils= .setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util= .ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.hive.metastore.Retr= yingRawStore.<init>(RetryingRawStore.java:62)
at org.apache.hadoop.hive= .metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
at org.apache.hadoop.hive.metast= ore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:413)
at org.apache.hadoop.hive= .metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:401)
at org.apache.hadoop.hive.met= astore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:439)
at org.apache.hadoop.hive= .metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:325)
= at org.apache.hadoop.hive.meta= store.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:285)
at org.apache.hadoop.hive= .metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:53)
at org.apache.hadoop.hiv= e.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:58)
at org.apache.hadoop.hive= .metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4102)
<= span style=3D"white-space:pre-wrap"> at org.apache.hadoop.hive.metas= tore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
... 28 more
Cau= sed by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.i= nvoke0(Native Method)
at sun.reflect.NativeMeth= odAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessor= Impl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Meth= od.invoke(Method.java:597)
= at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
at java.security.AccessCo= ntroller.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
at javax.jdo.JDOHelper.in= vokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
=
... 47 more
Caused by: java.lang.NoSuchFieldError: NUCLEUS_CONTEXT_LOADER
at org.datanucleus.NucleusCo= ntext.<clinit>(NucleusContext.java:105)
at org.datanucleus.api.jdo.JDO= PersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:39= 5)
at org.datanucleu= s.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOP= ersistenceManagerFactory.java:294)
at org.datanucleus.api.jd= o.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceM= anagerFactory.java:195)
... 55 more

FAILED: Execution Error, return code 1 from org.apache.= hadoop.hive.ql.exec.DDLTask
13/08/06 12:31:01 ERROR ql.Driver: FA= ILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DD= LTask
--089e0158b20a7d745c04e34ff1a2--