Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C6E3B19050 for ; Tue, 1 Mar 2016 05:06:11 +0000 (UTC) Received: (qmail 28265 invoked by uid 500); 1 Mar 2016 05:06:10 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 28215 invoked by uid 500); 1 Mar 2016 05:06:10 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 28205 invoked by uid 99); 1 Mar 2016 05:06:09 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2016 05:06:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 5AE8A1805CD for ; Tue, 1 Mar 2016 05:06:09 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.191 X-Spam-Level: * X-Spam-Status: No, score=1.191 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, T_KAM_HTML_FONT_INVALID=0.01, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=altiscale.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id VPIcyu6h2HkH for ; Tue, 1 Mar 2016 05:06:06 +0000 (UTC) Received: from mail-ob0-f180.google.com (mail-ob0-f180.google.com [209.85.214.180]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 6371F5F3DC for ; Tue, 1 Mar 2016 05:06:06 +0000 (UTC) Received: by mail-ob0-f180.google.com with SMTP id fz5so24636143obc.0 for ; Mon, 29 Feb 2016 21:06:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=altiscale.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to; bh=q5UhfTPFEDiB6zGG7Y/jksoYHwtuew2jpke7oggsLxc=; b=ifyoGAc/HrtjDPtStMSWQgd5BdCMWkX1jeAPSrYX52jaWKGGUgonWcPbBHngSM1yza B9ZK1aFw1drGw/GaByyXx5KdWxdcG+Z1vIQhiCK+HPUbYGYSWbmkAJjamvzdd3BC9UVX 9qu9IieUXESvs4ALWQYxZgVCrprxc9+BwzxhA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to; bh=q5UhfTPFEDiB6zGG7Y/jksoYHwtuew2jpke7oggsLxc=; b=MVHY4WblwjYlvVqMaLi306j+i/Gp/DghQtWSlOo5q5eKEb8a3/9PrEHMqvzmtCEjr7 NNP0dx2SkYbQZHPgCMYsXXFDFk1wQYSEe6kdY5RekbMsNrRqj9+h/EyTVz/Qpi3ZKSR9 4Gz93AK1Spu7ur7m0x5OcrATL9M93FyLR/d/XVkpzKXLsI6ExpdgnFy2otRVeJYK5gQs fWJNO63eaLMD/BK206a+ZUR7mMg96yhUSuWzOheo4GxtZ/4A3LSPxqgvS9XbgqqYGiIy Ccfmyq0mCuElmRF+FUkZHjgs04rwoY6P7f3PNNVG3OBl1nfDrebeg5PcV0OAiU1HGZCb uWMA== X-Gm-Message-State: AD7BkJIjXm+yEPFNMeQCFN7TgceXBJqU9fCtppH9MrnWH1aziQYQkVy8OqHuMzIyeRQq6uB/tcLZ67u9ZdaVbA== MIME-Version: 1.0 X-Received: by 10.182.196.104 with SMTP id il8mr14709395obc.71.1456808765740; Mon, 29 Feb 2016 21:06:05 -0800 (PST) Received: by 10.76.36.10 with HTTP; Mon, 29 Feb 2016 21:06:05 -0800 (PST) In-Reply-To: References: Date: Tue, 1 Mar 2016 10:36:05 +0530 Message-ID: Subject: Re: Sqoop Hcat Int partition error From: Biswajit Nayak To: user@hive.apache.org Content-Type: multipart/alternative; boundary=089e0149c636fce401052cf5b897 --089e0149c636fce401052cf5b897 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks Richa. The issue was suppose to be fixed in Hive 0.12 version as per the jira https://issues.apache.org/jira/browse/HIVE-7164. Even raised a ticket in sqoop jira [SQOOP-2840] for this . Thanks Biswa On Tue, Mar 1, 2016 at 9:56 AM, Richa Sharma wrote: > Hi, > > The values should still persist if partition column data type in Hive is = a > string. > > I am checking HCatalog documentation for support of int data type in > partition column. > > Cheers > Richa > > On Tue, Mar 1, 2016 at 3:06 PM, Biswajit Nayak > wrote: > >> Hi Richa, >> >> Thats a work around. But how to handle the columns with INT type. >> Changing the type will be the last option for me. >> >> Regards >> Biswa >> >> >> >> On Tue, Mar 1, 2016 at 9:31 AM, Richa Sharma > > wrote: >> >>> Hi Biswajit >>> >>> The answer is in the last line of the error message. Change the data >>> type of partition column to string in hive and try again. >>> >>> Hope it helps ! >>> >>> Richa >>> >>> 16/02/12 08:04:12 ERROR tool.ExportTool: Encountered IOException runnin= g export job: java.io.IOException: The table provided default.emp_details1 = uses unsupported partitioning key type for column salary : int. Only str= ing fields are allowed in partition columns in Catalog >>> >>> >>> On Tue, Mar 1, 2016 at 2:19 PM, Biswajit Nayak >>> wrote: >>> >>>> Hi All, >>>> >>>> I am trying to do a SQOOP export from hive( integer type partition) to >>>> mysql through HCAT and it fails with the following error. >>>> >>>> Versions:- >>>> >>>> Hadoop :- 2.7.1 >>>> Hive :- 1.2.0 >>>> Sqoop :- 1.4.5 >>>> >>>> Table in Hive :- >>>> >>>> >>>> hive> use default; >>>> OK >>>> Time taken: 0.028 seconds >>>> hive> describe emp_details1; >>>> OK >>>> id int >>>> name string >>>> deg string >>>> dept string >>>> salary int >>>> >>>> # Partition Information >>>> # col_name data_type comment >>>> >>>> salary int >>>> Time taken: 0.125 seconds, Fetched: 10 row(s) >>>> hive> >>>> >>>> hive> select * from emp_details1; >>>> OK >>>> 1201 gopal 50000 >>>> 1202 manisha 50000 >>>> 1203 kalil 50000 >>>> 1204 prasanth 50000 >>>> 1205 kranthi 50000 >>>> 1206 satish 50000 >>>> Time taken: 0.195 seconds, Fetched: 6 row(s) >>>> hive> >>>> >>>> >>>> Conf added to Hive metastore site.xml >>>> >>>> >>>> [alti-test-01@hdpnightly271-ci-91-services ~]$ grep -A5 -B2 -i "hive.m= etastore.integral.jdo.pushdown" /etc/hive-metastore/hive-site.xml >>>> >>>> >>>> hive.metastore.integral.jdo.pushdown >>>> TRUE >>>> >>>> >>>> >>>> [alti-test-01@hdpnightly271-ci-91-services ~]$ >>>> >>>> >>>> The issue remains same >>>> >>>> >>>> [alti-test-01@hdpnightly271-ci-91-services ~]$ /opt/sqoop-1.4.5/bin/sq= oop export --connect jdbc:mysql://localhost:3306/test --username hive --pas= sword ********* --table employee --hcatalog-database default --hcatalog-tab= le emp_details1 >>>> Warning: /opt/sqoop-1.4.5/bin/../../hbase does not exist! HBase import= s will fail. >>>> Please set $HBASE_HOME to the root of your HBase installation. >>>> Warning: /opt/sqoop-1.4.5/bin/../../accumulo does not exist! Accumulo = imports will fail. >>>> Please set $ACCUMULO_HOME to the root of your Accumulo installation. >>>> Warning: /opt/sqoop-1.4.5/bin/../../zookeeper does not exist! Accumulo= imports will fail. >>>> Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. >>>> 16/02/12 08:04:00 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5 >>>> 16/02/12 08:04:00 WARN tool.BaseSqoopTool: Setting your password on th= e command-line is insecure. Consider using -P instead. >>>> 16/02/12 08:04:00 INFO manager.MySQLManager: Preparing to use a MySQL = streaming resultset. >>>> 16/02/12 08:04:00 INFO tool.CodeGenTool: Beginning code generation >>>> 16/02/12 08:04:01 INFO manager.SqlManager: Executing SQL statement: SE= LECT t.* FROM `employee` AS t LIMIT 1 >>>> 16/02/12 08:04:01 INFO manager.SqlManager: Executing SQL statement: SE= LECT t.* FROM `employee` AS t LIMIT 1 >>>> 16/02/12 08:04:01 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /= opt/hadoop >>>> Note: /tmp/sqoop-alti-test-01/compile/1b0d4b1c30f167eb57ef488232ab49c8= /employee.java uses or overrides a deprecated API. >>>> Note: Recompile with -Xlint:deprecation for details. >>>> 16/02/12 08:04:07 INFO orm.CompilationManager: Writing jar file: /tmp/= sqoop-alti-test-01/compile/1b0d4b1c30f167eb57ef488232ab49c8/employee.jar >>>> 16/02/12 08:04:07 INFO mapreduce.ExportJobBase: Beginning export of em= ployee >>>> 16/02/12 08:04:08 INFO mapreduce.ExportJobBase: Configuring HCatalog f= or export job >>>> 16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Configuring HCatalog s= pecific details for job >>>> 16/02/12 08:04:08 INFO manager.SqlManager: Executing SQL statement: SE= LECT t.* FROM `employee` AS t LIMIT 1 >>>> 16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Database column names = projected : [id, name, deg, salary, dept] >>>> 16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Database column name -= info map : >>>> id : [Type : 4,Precision : 11,Scale : 0] >>>> name : [Type : 12,Precision : 20,Scale : 0] >>>> deg : [Type : 12,Precision : 20,Scale : 0] >>>> salary : [Type : 4,Precision : 11,Scale : 0] >>>> dept : [Type : 12,Precision : 10,Scale : 0] >>>> >>>> 16/02/12 08:04:10 INFO hive.metastore: Trying to connect to metastore = with URI thrift://hive-hdpnightly271-ci-91.test.altiscale.com:9083 >>>> 16/02/12 08:04:10 INFO hive.metastore: Connected to metastore. >>>> 16/02/12 08:04:11 INFO hcat.SqoopHCatUtilities: HCatalog full table sc= hema fields =3D [id, name, deg, dept, salary] >>>> 16/02/12 08:04:12 ERROR tool.ExportTool: Encountered IOException runni= ng export job: java.io.IOException: The table provided default.emp_details1= uses unsupported partitioning key type for column salary : int. Only st= ring fields are allowed in partition columns in Catalog >>>> >>>> >>>> Stuck with this issue. Any one had conquered this before. >>>> >>>> Regards >>>> Biswa >>>> >>>> >>> >> > --089e0149c636fce401052cf5b897 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks Richa.

The issue was suppose to = be fixed in Hive 0.12 version as per the jira https://issues.apache.org/jira/browse/HIVE-7= 164.=C2=A0

Even raised a ticket in sqoop jira= =C2=A0[SQOOP-2840]=C2=A0for this .=C2=A0

Thanks
Biswa
=





On Tue, Mar 1, 2016 at 9:5= 6 AM, Richa Sharma <mailtorichasharma@gmail.com> w= rote:
Hi,

T= he values should still persist if partition column data type in Hive is a s= tring.

I am checking HCatalog documentation for su= pport of int data type in partition column.

Cheers=
Richa
=

On Tue, Mar 1, 2016 at 3:06 PM, = Biswajit Nayak <biswajit@altiscale.com> wrote:
Hi Richa,=C2=A0

Thats a work around. But how to handle the columns with INT type. Changi= ng the type will be the last option for me.=C2=A0

= Regards
Biswa


=

On Tue, Mar 1, 20= 16 at 9:31 AM, Richa Sharma <mailtorichasharma@gmail.com>= wrote:
Hi Biswaj= it

The answer is in the last line of the error message. = Change the data type of partition column to string in hive and try again.

Hope it helps !
=

Richa
=
16/02/12 08:04:12 ERROR tool.ExportTool: E=
ncountered IOException running export job: java.io.IOException: The table p=
rovided default.emp_details1 uses unsupported  partitioning key type  for c=
olumn salary : int.  Only string fields are allowed in partition columns in=
 Catalog

On Tue, Mar 1, 2016 at 2:19 PM, Biswajit N= ayak <biswajit@altiscale.com> wrote:
Hi All,=C2=A0

I am trying to do a SQOOP export from hive( integer type partitio= n) to mysql through HCAT and it fails with the following error.=C2=A0

Versions:-

Hadoop :- =C2=A02.7.1
Hive =C2=A0 =C2=A0 =C2=A0:- =C2=A01.2.0
Sqoop =C2=A0 :- =C2=A01.4.5=

=

Table in= Hive :-


hive> use default;
OK
Time taken: 0.028 seconds
hive> describe emp_details1;
OK
id                      int                                        =20
name                    string                                     =20
deg                     string                                     =20
dept                    string                                     =20
salary                  int                                        =20

# Partition Information     =20
# col_name              data_type               comment            =20

salary                  int                                        =20
Time taken: 0.125 seconds, Fetched: 10 row(s)
hive>=20

hive> select * from emp_details1;
OK
1201    gopal           50000
1202    manisha         50000
1203    kalil           50000
1204    prasanth        50000
1205    kranthi         50000
1206    satish          50000
Time taken: 0.195 seconds, Fetched: 6 row(s)
hive>=20

= Conf added to Hive metastore site.xml


[alti-test-01@hdpnightly271-ci-91-services ~]$ grep -A5 -B2 -i "hive.m=
etastore.integral.jdo.pushdown" /etc/hive-metastore/hive-site.xml=20
    </property>
    <property>
        <name>hive.metastore.integral.jdo.pushdown</name>
        <value>TRUE</value>
    </property>

</configuration>
[alti-test-01@hdpnightly271-ci-91-services ~]$=20

= The issue remains same


[alti-test-01@hdpnightly271-ci-91-services ~]$ /opt/sqoop-1.4.5/bin/sqoop e=
xport --connect jdbc:mysql://localhost:3306/test --username hive --password=
 ********* --table employee --hcatalog-database default --hcatalog-table em=
p_details1
Warning: /opt/sqoop-1.4.5/bin/../../hbase does not exist! HBase imports wil=
l fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/sqoop-1.4.5/bin/../../accumulo does not exist! Accumulo impor=
ts will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/sqoop-1.4.5/bin/../../zookeeper does not exist! Accumulo impo=
rts will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
16/02/12 08:04:00 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5
16/02/12 08:04:00 WARN tool.BaseSqoopTool: Setting your password on the com=
mand-line is insecure. Consider using -P instead.
16/02/12 08:04:00 INFO manager.MySQLManager: Preparing to use a MySQL strea=
ming resultset.
16/02/12 08:04:00 INFO tool.CodeGenTool: Beginning code generation
16/02/12 08:04:01 INFO manager.SqlManager: Executing SQL statement: SELECT =
t.* FROM `employee` AS t LIMIT 1
16/02/12 08:04:01 INFO manager.SqlManager: Executing SQL statement: SELECT =
t.* FROM `employee` AS t LIMIT 1
16/02/12 08:04:01 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/h=
adoop
Note: /tmp/sqoop-alti-test-01/compile/1b0d4b1c30f167eb57ef488232ab49c8/empl=
oyee.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/02/12 08:04:07 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop=
-alti-test-01/compile/1b0d4b1c30f167eb57ef488232ab49c8/employee.jar
16/02/12 08:04:07 INFO mapreduce.ExportJobBase: Beginning export of employe=
e
16/02/12 08:04:08 INFO mapreduce.ExportJobBase: Configuring HCatalog for ex=
port job
16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Configuring HCatalog specif=
ic details for job
16/02/12 08:04:08 INFO manager.SqlManager: Executing SQL statement: SELECT =
t.* FROM `employee` AS t LIMIT 1
16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Database column names proje=
cted : [id, name, deg, salary, dept]
16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Database column name - info=
 map :
    id : [Type : 4,Precision : 11,Scale : 0]
    name : [Type : 12,Precision : 20,Scale : 0]
    deg : [Type : 12,Precision : 20,Scale : 0]
    salary : [Type : 4,Precision : 11,Scale : 0]
    dept : [Type : 12,Precision : 10,Scale : 0]

16/02/12 08:04:10 INFO hive.metastore: Trying to connect to metastore with =
URI thrift://hive-hdpnightly271-ci-91.test.altiscale.com:9083
16/02/12 08:04:10 INFO hive.metastore: Connected to metastore.
16/02/12 08:04:11 INFO hcat.SqoopHCatUtilities: HCatalog full table schema =
fields =3D [id, name, deg, dept, salary]
16/02/12 08:04:12 ERROR tool.ExportTool: Encountered IOException running ex=
port job: java.io.IOException: The table provided default.emp_details1 uses=
 unsupported  partitioning key type  for column salary : int.  Only string =
fields are allowed in partition columns in Catalog







--089e0149c636fce401052cf5b897--