Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 48D5B18F20 for ; Tue, 1 Mar 2016 04:01:11 +0000 (UTC) Received: (qmail 12976 invoked by uid 500); 1 Mar 2016 04:01:06 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 12880 invoked by uid 500); 1 Mar 2016 04:01:06 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 12870 invoked by uid 99); 1 Mar 2016 04:01:06 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2016 04:01:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 36F871A0CF6 for ; Tue, 1 Mar 2016 04:01:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.19 X-Spam-Level: * X-Spam-Status: No, score=1.19 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id nM0iq4yXkIwf for ; Tue, 1 Mar 2016 04:01:03 +0000 (UTC) Received: from mail-wm0-f45.google.com (mail-wm0-f45.google.com [74.125.82.45]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 519C05F1B9 for ; Tue, 1 Mar 2016 04:01:03 +0000 (UTC) Received: by mail-wm0-f45.google.com with SMTP id p65so16098940wmp.0 for ; Mon, 29 Feb 2016 20:01:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to; bh=dVX4fwiRIw0sQ9P0IZ1Jq4pp4NF1shK0JyginFN6nOY=; b=IKK9JMS9TFtq1kWlMmcZmyy5mlKQw0TMJefiSV7S4ZrakJ7WSf26KArR08dodO1wDJ BA4HObwCzQLSBkhtcv/mEqiO5asOJumdUa4RFPfLxfbyYO+GBP+fk6mQg7TuEfpYPmp4 oVpTh9DS6H925VhgNRyq1Ux0tc+eaegV0JeONkWI/l/f58AHhe/pVcWfZ5Qq+yXCzteL chENpcVGlnV+xZyNzq9+zcvwgftTFXPCsSrRponiRRF7IWVAuy2+J8dEcTu0Grpl/qnr k2PDTM3wKh2yKwzz81lk24Y5mGzgjhDLPwn8FrLbJoNlCxFLFInY6u54KG5YTBOSJKLC zd/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to; bh=dVX4fwiRIw0sQ9P0IZ1Jq4pp4NF1shK0JyginFN6nOY=; b=N1t+Dh2/SrzIc2jINa1sNMp2tUg1dUzopGv34vak/HKpzLgGZr0AEQCEMUv884Gw+h R/KyujDaMDayEzCZjLedl56f+LkNDC8/qYuKsRF+kjpz57+UNgcsnwoVQGUXdJJ11yRj RYH2FUjNfTo3181ldSFFZZJXiFDaCMWdGrjI5jLrPZXqOM+2fh/eAHrGc8Vk5A12uecF 7dAC+mW08rwc3nyWkhJP4tfpGJiWclA/zCWx/dq3AiB3x2K65UfeTcPMSZypnXZ6idAz LXlo1UqSL9KdJRLwcAQg9h9w6dXqd/GLhv3Ur8hvi/I1fVN1obLni5jsJIYE0W00an8M RFrw== X-Gm-Message-State: AD7BkJLDNR75YMmeWj36xksq6xIupef6ds7dzLslZD2TjvMQ6TnOKIanJU+H1GPeH58mDaW4wpb3Am3PipwUfw== MIME-Version: 1.0 X-Received: by 10.194.63.242 with SMTP id j18mr21372259wjs.114.1456804863026; Mon, 29 Feb 2016 20:01:03 -0800 (PST) Received: by 10.28.24.69 with HTTP; Mon, 29 Feb 2016 20:01:02 -0800 (PST) In-Reply-To: References: Date: Tue, 1 Mar 2016 15:01:02 +1100 Message-ID: Subject: Re: Sqoop Hcat Int partition error From: Richa Sharma To: user@hive.apache.org Content-Type: multipart/alternative; boundary=047d7ba982fe5e2ba3052cf4d057 --047d7ba982fe5e2ba3052cf4d057 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Biswajit The answer is in the last line of the error message. Change the data type of partition column to string in hive and try again. Hope it helps ! Richa 16/02/12 08:04:12 ERROR tool.ExportTool: Encountered IOException running export job: java.io.IOException: The table provided default.emp_details1 uses unsupported partitioning key type for column salary : int. Only string fields are allowed in partition columns in Catalog On Tue, Mar 1, 2016 at 2:19 PM, Biswajit Nayak wrote: > Hi All, > > I am trying to do a SQOOP export from hive( integer type partition) to > mysql through HCAT and it fails with the following error. > > Versions:- > > Hadoop :- 2.7.1 > Hive :- 1.2.0 > Sqoop :- 1.4.5 > > Table in Hive :- > > > hive> use default; > OK > Time taken: 0.028 seconds > hive> describe emp_details1; > OK > id int > name string > deg string > dept string > salary int > > # Partition Information > # col_name data_type comment > > salary int > Time taken: 0.125 seconds, Fetched: 10 row(s) > hive> > > hive> select * from emp_details1; > OK > 1201 gopal 50000 > 1202 manisha 50000 > 1203 kalil 50000 > 1204 prasanth 50000 > 1205 kranthi 50000 > 1206 satish 50000 > Time taken: 0.195 seconds, Fetched: 6 row(s) > hive> > > > Conf added to Hive metastore site.xml > > > [alti-test-01@hdpnightly271-ci-91-services ~]$ grep -A5 -B2 -i "hive.meta= store.integral.jdo.pushdown" /etc/hive-metastore/hive-site.xml > > > hive.metastore.integral.jdo.pushdown > TRUE > > > > [alti-test-01@hdpnightly271-ci-91-services ~]$ > > > The issue remains same > > > [alti-test-01@hdpnightly271-ci-91-services ~]$ /opt/sqoop-1.4.5/bin/sqoop= export --connect jdbc:mysql://localhost:3306/test --username hive --passwo= rd ********* --table employee --hcatalog-database default --hcatalog-table = emp_details1 > Warning: /opt/sqoop-1.4.5/bin/../../hbase does not exist! HBase imports w= ill fail. > Please set $HBASE_HOME to the root of your HBase installation. > Warning: /opt/sqoop-1.4.5/bin/../../accumulo does not exist! Accumulo imp= orts will fail. > Please set $ACCUMULO_HOME to the root of your Accumulo installation. > Warning: /opt/sqoop-1.4.5/bin/../../zookeeper does not exist! Accumulo im= ports will fail. > Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. > 16/02/12 08:04:00 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5 > 16/02/12 08:04:00 WARN tool.BaseSqoopTool: Setting your password on the c= ommand-line is insecure. Consider using -P instead. > 16/02/12 08:04:00 INFO manager.MySQLManager: Preparing to use a MySQL str= eaming resultset. > 16/02/12 08:04:00 INFO tool.CodeGenTool: Beginning code generation > 16/02/12 08:04:01 INFO manager.SqlManager: Executing SQL statement: SELEC= T t.* FROM `employee` AS t LIMIT 1 > 16/02/12 08:04:01 INFO manager.SqlManager: Executing SQL statement: SELEC= T t.* FROM `employee` AS t LIMIT 1 > 16/02/12 08:04:01 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt= /hadoop > Note: /tmp/sqoop-alti-test-01/compile/1b0d4b1c30f167eb57ef488232ab49c8/em= ployee.java uses or overrides a deprecated API. > Note: Recompile with -Xlint:deprecation for details. > 16/02/12 08:04:07 INFO orm.CompilationManager: Writing jar file: /tmp/sqo= op-alti-test-01/compile/1b0d4b1c30f167eb57ef488232ab49c8/employee.jar > 16/02/12 08:04:07 INFO mapreduce.ExportJobBase: Beginning export of emplo= yee > 16/02/12 08:04:08 INFO mapreduce.ExportJobBase: Configuring HCatalog for = export job > 16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Configuring HCatalog spec= ific details for job > 16/02/12 08:04:08 INFO manager.SqlManager: Executing SQL statement: SELEC= T t.* FROM `employee` AS t LIMIT 1 > 16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Database column names pro= jected : [id, name, deg, salary, dept] > 16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Database column name - in= fo map : > id : [Type : 4,Precision : 11,Scale : 0] > name : [Type : 12,Precision : 20,Scale : 0] > deg : [Type : 12,Precision : 20,Scale : 0] > salary : [Type : 4,Precision : 11,Scale : 0] > dept : [Type : 12,Precision : 10,Scale : 0] > > 16/02/12 08:04:10 INFO hive.metastore: Trying to connect to metastore wit= h URI thrift://hive-hdpnightly271-ci-91.test.altiscale.com:9083 > 16/02/12 08:04:10 INFO hive.metastore: Connected to metastore. > 16/02/12 08:04:11 INFO hcat.SqoopHCatUtilities: HCatalog full table schem= a fields =3D [id, name, deg, dept, salary] > 16/02/12 08:04:12 ERROR tool.ExportTool: Encountered IOException running = export job: java.io.IOException: The table provided default.emp_details1 us= es unsupported partitioning key type for column salary : int. Only strin= g fields are allowed in partition columns in Catalog > > > Stuck with this issue. Any one had conquered this before. > > Regards > Biswa > > --047d7ba982fe5e2ba3052cf4d057 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Biswajit

The answer is in the last l= ine of the error message. Change the data type of partition column to strin= g in hive and try again.

Hope it helps !

Richa
16/02/12 08:04:12 ERROR tool.ExportTool: Encountered IOException r=
unning export job: java.io.IOException: The table provided default.emp_deta=
ils1 uses unsupported  partitioning key type  for column salary : int.  Onl=
y string fields are allowed in partition columns in Catalog

On Tue, = Mar 1, 2016 at 2:19 PM, Biswajit Nayak <biswajit@altiscale.com>= ; wrote:
Hi All,=C2=A0

I am trying to do a SQOOP expo= rt from hive( integer type partition) to mysql through HCAT and it fails wi= th the following error.=C2=A0

= Versions:-

Hadoop :- =C2=A02.7.= 1
Hive =C2=A0 =C2= =A0 =C2=A0:- =C2=A01.2.0
Sqoop =C2=A0 :- =C2=A01.4.5

Table in Hive :-


hive> use default;
OK
Time taken: 0.028 seconds
hive> describe emp_details1;
OK
id                      int                                        =20
name                    string                                     =20
deg                     string                                     =20
dept                    string                                     =20
salary                  int                                        =20

# Partition Information     =20
# col_name              data_type               comment            =20

salary                  int                                        =20
Time taken: 0.125 seconds, Fetched: 10 row(s)
hive>=20

hive> select * from emp_details1;
OK
1201    gopal           50000
1202    manisha         50000
1203    kalil           50000
1204    prasanth        50000
1205    kranthi         50000
1206    satish          50000
Time taken: 0.195 seconds, Fetched: 6 row(s)
hive>=20

= Conf added to Hive metastore site.xml


[alti-test-01@hdpnightly271-ci-91-services ~]$ grep -A5 -B2 -i "hive.m=
etastore.integral.jdo.pushdown" /etc/hive-metastore/hive-site.xml=20
    </property>
    <property>
        <name>hive.metastore.integral.jdo.pushdown</name>
        <value>TRUE</value>
    </property>

</configuration>
[alti-test-01@hdpnightly271-ci-91-services ~]$=20

= The issue remains same


[alti-test-01@hdpnightly271-ci-91-services ~]$ /opt/sqoop-1.4.5/bin/sqoop e=
xport --connect jdbc:mysql://localhost:3306/test --username hive --password=
 ********* --table employee --hcatalog-database default --hcatalog-table em=
p_details1
Warning: /opt/sqoop-1.4.5/bin/../../hbase does not exist! HBase imports wil=
l fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/sqoop-1.4.5/bin/../../accumulo does not exist! Accumulo impor=
ts will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/sqoop-1.4.5/bin/../../zookeeper does not exist! Accumulo impo=
rts will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
16/02/12 08:04:00 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5
16/02/12 08:04:00 WARN tool.BaseSqoopTool: Setting your password on the com=
mand-line is insecure. Consider using -P instead.
16/02/12 08:04:00 INFO manager.MySQLManager: Preparing to use a MySQL strea=
ming resultset.
16/02/12 08:04:00 INFO tool.CodeGenTool: Beginning code generation
16/02/12 08:04:01 INFO manager.SqlManager: Executing SQL statement: SELECT =
t.* FROM `employee` AS t LIMIT 1
16/02/12 08:04:01 INFO manager.SqlManager: Executing SQL statement: SELECT =
t.* FROM `employee` AS t LIMIT 1
16/02/12 08:04:01 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/h=
adoop
Note: /tmp/sqoop-alti-test-01/compile/1b0d4b1c30f167eb57ef488232ab49c8/empl=
oyee.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/02/12 08:04:07 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop=
-alti-test-01/compile/1b0d4b1c30f167eb57ef488232ab49c8/employee.jar
16/02/12 08:04:07 INFO mapreduce.ExportJobBase: Beginning export of employe=
e
16/02/12 08:04:08 INFO mapreduce.ExportJobBase: Configuring HCatalog for ex=
port job
16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Configuring HCatalog specif=
ic details for job
16/02/12 08:04:08 INFO manager.SqlManager: Executing SQL statement: SELECT =
t.* FROM `employee` AS t LIMIT 1
16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Database column names proje=
cted : [id, name, deg, salary, dept]
16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Database column name - info=
 map :
    id : [Type : 4,Precision : 11,Scale : 0]
    name : [Type : 12,Precision : 20,Scale : 0]
    deg : [Type : 12,Precision : 20,Scale : 0]
    salary : [Type : 4,Precision : 11,Scale : 0]
    dept : [Type : 12,Precision : 10,Scale : 0]

16/02/12 08:04:10 INFO hive.metastore: Trying to connect to metastore with =
URI thrift://hive-hdpnightly271-ci-91.test.altiscale.com:9083
16/02/12 08:04:10 INFO hive.metastore: Connected to metastore.
16/02/12 08:04:11 INFO hcat.SqoopHCatUtilities: HCatalog full table schema =
fields =3D [id, name, deg, dept, salary]
16/02/12 08:04:12 ERROR tool.ExportTool: Encountered IOException running ex=
port job: java.io.IOException: The table provided default.emp_details1 uses=
 unsupported  partitioning key type  for column salary : int.  Only string =
fields are allowed in partition columns in Catalog




--047d7ba982fe5e2ba3052cf4d057--