hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Biswajit Nayak <biswa...@altiscale.com>
Subject Re: Sqoop Hcat Int partition error
Date Tue, 01 Mar 2016 04:06:41 GMT
Hi Richa,

Thats a work around. But how to handle the columns with INT type. Changing
the type will be the last option for me.

Regards
Biswa



On Tue, Mar 1, 2016 at 9:31 AM, Richa Sharma <mailtorichasharma@gmail.com>
wrote:

> Hi Biswajit
>
> The answer is in the last line of the error message. Change the data type
> of partition column to string in hive and try again.
>
> Hope it helps !
>
> Richa
>
> 16/02/12 08:04:12 ERROR tool.ExportTool: Encountered IOException running export job:
java.io.IOException: The table provided default.emp_details1 uses unsupported  partitioning
key type  for column salary : int.  Only string fields are allowed in partition columns in
Catalog
>
>
> On Tue, Mar 1, 2016 at 2:19 PM, Biswajit Nayak <biswajit@altiscale.com>
> wrote:
>
>> Hi All,
>>
>> I am trying to do a SQOOP export from hive( integer type partition) to
>> mysql through HCAT and it fails with the following error.
>>
>> Versions:-
>>
>> Hadoop :-  2.7.1
>> Hive      :-  1.2.0
>> Sqoop   :-  1.4.5
>>
>> Table in Hive :-
>>
>>
>> hive> use default;
>> OK
>> Time taken: 0.028 seconds
>> hive> describe emp_details1;
>> OK
>> id                      int
>> name                    string
>> deg                     string
>> dept                    string
>> salary                  int
>>
>> # Partition Information
>> # col_name              data_type               comment
>>
>> salary                  int
>> Time taken: 0.125 seconds, Fetched: 10 row(s)
>> hive>
>>
>> hive> select * from emp_details1;
>> OK
>> 1201    gopal           50000
>> 1202    manisha         50000
>> 1203    kalil           50000
>> 1204    prasanth        50000
>> 1205    kranthi         50000
>> 1206    satish          50000
>> Time taken: 0.195 seconds, Fetched: 6 row(s)
>> hive>
>>
>>
>> Conf added to Hive metastore site.xml
>>
>>
>> [alti-test-01@hdpnightly271-ci-91-services ~]$ grep -A5 -B2 -i "hive.metastore.integral.jdo.pushdown"
/etc/hive-metastore/hive-site.xml
>>     </property>
>>     <property>
>>         <name>hive.metastore.integral.jdo.pushdown</name>
>>         <value>TRUE</value>
>>     </property>
>>
>> </configuration>
>> [alti-test-01@hdpnightly271-ci-91-services ~]$
>>
>>
>> The issue remains same
>>
>>
>> [alti-test-01@hdpnightly271-ci-91-services ~]$ /opt/sqoop-1.4.5/bin/sqoop export
--connect jdbc:mysql://localhost:3306/test --username hive --password ********* --table employee
--hcatalog-database default --hcatalog-table emp_details1
>> Warning: /opt/sqoop-1.4.5/bin/../../hbase does not exist! HBase imports will fail.
>> Please set $HBASE_HOME to the root of your HBase installation.
>> Warning: /opt/sqoop-1.4.5/bin/../../accumulo does not exist! Accumulo imports will
fail.
>> Please set $ACCUMULO_HOME to the root of your Accumulo installation.
>> Warning: /opt/sqoop-1.4.5/bin/../../zookeeper does not exist! Accumulo imports will
fail.
>> Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
>> 16/02/12 08:04:00 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5
>> 16/02/12 08:04:00 WARN tool.BaseSqoopTool: Setting your password on the command-line
is insecure. Consider using -P instead.
>> 16/02/12 08:04:00 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
>> 16/02/12 08:04:00 INFO tool.CodeGenTool: Beginning code generation
>> 16/02/12 08:04:01 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM
`employee` AS t LIMIT 1
>> 16/02/12 08:04:01 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM
`employee` AS t LIMIT 1
>> 16/02/12 08:04:01 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop
>> Note: /tmp/sqoop-alti-test-01/compile/1b0d4b1c30f167eb57ef488232ab49c8/employee.java
uses or overrides a deprecated API.
>> Note: Recompile with -Xlint:deprecation for details.
>> 16/02/12 08:04:07 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-alti-test-01/compile/1b0d4b1c30f167eb57ef488232ab49c8/employee.jar
>> 16/02/12 08:04:07 INFO mapreduce.ExportJobBase: Beginning export of employee
>> 16/02/12 08:04:08 INFO mapreduce.ExportJobBase: Configuring HCatalog for export job
>> 16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Configuring HCatalog specific details
for job
>> 16/02/12 08:04:08 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM
`employee` AS t LIMIT 1
>> 16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Database column names projected :
[id, name, deg, salary, dept]
>> 16/02/12 08:04:08 INFO hcat.SqoopHCatUtilities: Database column name - info map :
>>     id : [Type : 4,Precision : 11,Scale : 0]
>>     name : [Type : 12,Precision : 20,Scale : 0]
>>     deg : [Type : 12,Precision : 20,Scale : 0]
>>     salary : [Type : 4,Precision : 11,Scale : 0]
>>     dept : [Type : 12,Precision : 10,Scale : 0]
>>
>> 16/02/12 08:04:10 INFO hive.metastore: Trying to connect to metastore with URI thrift://hive-hdpnightly271-ci-91.test.altiscale.com:9083
>> 16/02/12 08:04:10 INFO hive.metastore: Connected to metastore.
>> 16/02/12 08:04:11 INFO hcat.SqoopHCatUtilities: HCatalog full table schema fields
= [id, name, deg, dept, salary]
>> 16/02/12 08:04:12 ERROR tool.ExportTool: Encountered IOException running export job:
java.io.IOException: The table provided default.emp_details1 uses unsupported  partitioning
key type  for column salary : int.  Only string fields are allowed in partition columns in
Catalog
>>
>>
>> Stuck with this issue. Any one had conquered this before.
>>
>> Regards
>> Biswa
>>
>>
>

Mime
View raw message