Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8FEB399F7 for ; Tue, 25 Oct 2011 02:49:40 +0000 (UTC) Received: (qmail 95117 invoked by uid 500); 25 Oct 2011 02:49:39 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 95034 invoked by uid 500); 25 Oct 2011 02:49:38 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 95025 invoked by uid 99); 25 Oct 2011 02:49:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Oct 2011 02:49:38 +0000 X-ASF-Spam-Status: No, hits=2.6 required=5.0 tests=FREEMAIL_FROM,FROM_LOCAL_NOVOWEL,HK_RANDOM_ENVFROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rckclmbr@gmail.com designates 74.125.82.54 as permitted sender) Received: from [74.125.82.54] (HELO mail-ww0-f54.google.com) (74.125.82.54) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Oct 2011 02:49:30 +0000 Received: by wwe5 with SMTP id 5so37287wwe.23 for ; Mon, 24 Oct 2011 19:49:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=i1FKGaBi9tKhUm9IZ6Q+Ylt6HgnOURCqOEGJ/azEwEg=; b=Fw7by6whKjOzhPw+ZOJ6EEo5wiBynQF+HXZ1kAA9fN89V/XatLNMymvmaNH+3BqMi5 Yr7EaCaScB4c+ZY2d3AeG2eqm4tZA4mSq9sXLF+OwDTE7HL7myIETfoTSNP08SaSxKPo CCcxeY3FxGp0Ak0oEql55LA0bWWnIGpFBKta4= MIME-Version: 1.0 Received: by 10.216.221.163 with SMTP id r35mr4195335wep.21.1319510949135; Mon, 24 Oct 2011 19:49:09 -0700 (PDT) Received: by 10.216.169.10 with HTTP; Mon, 24 Oct 2011 19:49:09 -0700 (PDT) Date: Mon, 24 Oct 2011 20:49:09 -0600 Message-ID: Subject: Hive throwing exception when using Sqoop From: Joshua Braegger To: user@hive.apache.org Content-Type: multipart/alternative; boundary=0016e65a0986668a5d04b016955e X-Virus-Checked: Checked by ClamAV on apache.org --0016e65a0986668a5d04b016955e Content-Type: text/plain; charset=ISO-8859-1 Hello, I am trying to use Sqoop to import data into Hive, and am running the following command: sqoop import --connect 'jdbc:mysql://dbhost/dbname' --username xxuser --table xxtable --password xxpassword --hive-import --direct and am receiving the following exception from Hive: 11/10/24 20:36:31 INFO hive.HiveImport: Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.security.UserGroupInformation.login(Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/security/UserGroupInformation; 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.hive.shims.Hadoop20Shims.getUGIForConf(Hadoop20Shims.java:448) 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator.setConf(HadoopDefaultAuthenticator.java:51) 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:222) 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:241) 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:466) 11/10/24 20:36:31 INFO hive.HiveImport: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 11/10/24 20:36:31 INFO hive.HiveImport: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 11/10/24 20:36:31 INFO hive.HiveImport: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 11/10/24 20:36:31 INFO hive.HiveImport: at java.lang.reflect.Method.invoke(Method.java:616) 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.util.RunJar.main(RunJar.java:186) I've seen others with the same issue, but no mention of a resolution. I'm fairly certain this is an issue with Hive, but haven't been able to reproduce the issue directly with Hive... only using Sqoop. I'm using the Cloudera-distributed release of all 3 applications (downloaded from https://ccp.cloudera.com/display/SUPPORT/CDH3+Downloadable+Tarballs). The specific versions are: * hadoop-0.20.2-cdh3u2 * hive-0.7.1-cdh3u2 * sqoop-1.3.0-cdh3u2 I have seen that others are using hive 0.7.1 with hadoop 0.20.2, but can't figure out why it's working at all for them. In the file "shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java", getUGIForConf is defined as: public UserGroupInformation getUGIForConf(Configuration conf) throws LoginException { UserGroupInformation ugi = UnixUserGroupInformation.readFromConf(conf, UnixUserGroupInformation.UGI_PROPERTY_NAME); if(ugi == null) { ugi = UserGroupInformation.login(conf); } return ugi; } and the offending line is "ugi = UserGroupInformation.login(conf)". Looking at the source for hadoop's UserGroupInformation.java, there is no "login()" method defined -- the only ones I could see working are "getLoginUser" and "getCurrentUser". Attempting to change the "login" to one of these fails immediately, because 0.20.1 (which Hive is built with) doesn't contain these methods, but 0.20.2 does. Looking at the shems, there's no support for sub-versions, only major version (.19, .20, etc), so I'm not sure how trivial it would be to break it apart into minor releases. Given that I know these two versions should be pretty compatible with each other (since Cloudera is distributing them together), they are stable releases, and I haven't seen others mentioning a problem, does anyone know 1) if this is actually a problem and 2) Is there a workaround or simple fix that can be applied. Thanks, -Josh Braegger --0016e65a0986668a5d04b016955e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello,

I am trying to use Sqoop to import data into Hive, and am running th= e following command:

sqoop import --connect 'jdbc:mysql://dbhost/dbname&= #39; --username xxuser --table xxtable --password xxpassword --hive-import = --direct

and am receiving the following exception = from Hive:

11/10/24 20:36:31 INFO hive.HiveImport: Exception = in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.s= ecurity.UserGroupInformation.login(Lorg/apache/hadoop/conf/Configuration;)L= org/apache/hadoop/security/UserGroupInformation;
11/10/24 20:36:31 INFO hive.HiveImport:=A0 at org.apache.hadoop.hive.shims.Hadoop20Shims.getUGIFor= Conf(Hadoop20Shims.java:448)
11/10/24 20:36:31 INFO hive.HiveImpo= rt:=A0 at org.apache.hadoop.= hive.ql.security.HadoopDefaultAuthenticator.setConf(HadoopDefaultAuthentica= tor.java:51)
11/10/24 20:36:31 INFO hive.HiveImport:=A0 at org.apache.hadoop.util.ReflectionUtils.setConf(Refle= ctionUtils.java:62)
11/10/24 20:36:31 INFO hive.HiveImport:=A0 at org.apache.hadoop.util.Refl= ectionUtils.newInstance(ReflectionUtils.java:117)
11/10/24 20:36:31 INFO hive.HiveImport:=A0 at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuth= enticator(HiveUtils.java:222)
11/10/24 20:36:31 INFO hive.HiveImp= ort:=A0 at org.apache.hadoop= .hive.ql.session.SessionState.start(SessionState.java:241)
11/10/24 20:36:31 INFO hive.HiveImport:=A0 at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.= java:466)
11/10/24 20:36:31 INFO hive.HiveImport:=A0 at sun.reflect.NativeMethodAccessorImp= l.invoke0(Native Method)
11/10/24 20:36:31 INFO hive.HiveImport:=A0 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMe= thodAccessorImpl.java:57)
11/10/24 20:36:31 INFO hive.HiveImport:= =A0 at sun.reflect.Delegatin= gMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11/10/24 20:36:31 INFO hive.HiveImport:=A0 at java.lang.reflect.Method.invoke(Method.java:616)
11/10/24 20:36:31 INFO hive.HiveImport:=A0 at org.apache.hadoop.util.RunJar.main(RunJar.java:186= )

I've seen others with the same issue, but no = mention of a resolution. =A0I'm fairly certain this is an issue with Hi= ve, but haven't been able to reproduce the issue directly with Hive... = only using Sqoop. =A0I'm using the Cloudera-distributed release of all = 3 applications (downloaded from=A0https://ccp.cloudera.com/display/SUPPORT/CDH3+Downloadable+= Tarballs). =A0The specific versions are:

*=A0hadoop-0.20.2-cdh3u2
*=A0hive-0.7.1-cdh3u= 2
*=A0sqoop-1.3.0-cdh3u2

I have seen tha= t others are using hive 0.7.1 with hadoop 0.20.2, but can't figure out = why it's working at all for them. =A0In the file "shims/src/0.20/j= ava/org/apache/hadoop/hive/shims/Hadoop20Shims.java", getUGIForConf is= defined as:

=A0 public UserGroupInformation getUGIForConf(Conf= iguration conf) throws LoginException {
=A0 =A0 UserGroupInformat= ion ugi =3D
=A0 =A0 =A0 UnixUserGroupInformation.readFromConf(con= f, UnixUserGroupInformation.UGI_PROPERTY_NAME);
=A0 =A0 if(ugi =3D=3D null) {
=A0 =A0 =A0 ugi =3D UserGroupI= nformation.login(conf);
=A0 =A0 }
=A0 =A0 return ugi;
=A0 }

and the offending line is &qu= ot;ugi =3D UserGroupInformation.login(conf)". =A0Looking at the source= for hadoop's UserGroupInformation.java, there is no "login()"= ; method defined -- the only ones I could see working are "getLoginUse= r" and "getCurrentUser". =A0Attempting to change the "l= ogin" to one of these fails immediately, because 0.20.1 (which Hive is= built with) doesn't contain these methods, but 0.20.2 does. =A0Looking= at the shems, there's no support for sub-versions, only major version = (.19, .20, etc), so I'm not sure how trivial it would be to break it ap= art into minor releases.

Given that I know these two versions should be pretty c= ompatible with each other (since Cloudera is distributing them together), t= hey are stable releases, and I haven't seen others mentioning a problem= , does anyone know 1) if this is actually a problem and 2) Is there a worka= round or simple fix that can be applied.

Thanks,
-Josh Braegger
--0016e65a0986668a5d04b016955e--