Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Tue, 23 May 2017 07:29:04 +0000 (UTC)
From: "Remus Rusanu (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.13072020.1494856474000.269800.1495524544069@Atlassian.JIRA>
In-Reply-To: <JIRA.13072020.1494856474000@Atlassian.JIRA>
References: <JIRA.13072020.1494856474000@Atlassian.JIRA> <JIRA.13072020.1494856474868@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HIVE-16667) PostgreSQL metastore handling of
 CLOB types for COLUMNS_V2.TYPE_NAME and other field is incorrect
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 23 May 2017 07:29:09 -0000


    [ https://issues.apache.org/jira/browse/HIVE-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020759#comment-16020759 ] 

Remus Rusanu commented on HIVE-16667:
-------------------------------------

I don't see how your patch will address the problem of mixing  pre v3.0 upgraded tables and post v.30 created one. the new ones will store the {{TYPE_NAME}} as a LOB handle, while the upgraded ones will have the inlined string value.

Can we explore instead reverting the column type mapping to {{VARCHAR}} in the package.jdo, (or {{LONGVARCHAR}} if need be) instead of {{CLOB}}? Keep the metastore upgrade scripts, change the underlying storage tyopes to respective large types, but have JDO map them to String, not Clob. On my testing with PG, this works correctly on all cases I tested (my repro, your large serde DDL from HIVE-12274), and should handle upgrade correctly. But I did not test this with other metastore engines, Oracle, MySQL etc.

> PostgreSQL metastore handling of CLOB types for COLUMNS_V2.TYPE_NAME and other field is incorrect
> -------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-16667
>                 URL: https://issues.apache.org/jira/browse/HIVE-16667
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Remus Rusanu
>            Assignee: Naveen Gangam
>         Attachments: HIVE-16667.patch, HiveCLIOutput.txt, PostgresDBOutput.txt
>
>
> The CLOB JDO type introduced with HIVE-12274 does not work correctly with PostgreSQL. The value is written out-of-band and the LOB handle is written,as an INT, into the table. SELECTs return the INT value, which should had been read via the {{lo_get}} PG built-in, and then cast into string.
> Furthermore, the behavior is different between fields upgraded from earlier metastore versions (they retain their string storage) vs. values inserted after the upgrade (inserted as LOB roots).
> Teh code in {{MetasoreDirectSql.getPartitionsFromPartitionIds/extractSqlClob}} expects the underlying JDO/Datanucleus to map the column to a {{Clob}} but that does not happen, the value is a Java String containing the int which is the LOB root saved by PG.
> This manifests at runtime with errors like:
> {code}
> hive> select * from srcpart;
> Failed with exception java.io.IOException:java.lang.IllegalArgumentException: Error: type expected at the position 0 of '24030:24031' but '24030' is found.
> {code}
> the 24030:24031 should be 'string:string'.
> repro:
> {code}
> CREATE TABLE srcpart (key STRING COMMENT 'default', value STRING COMMENT 'default') PARTITIONED BY (ds STRING, hr STRING) STORED AS TEXTFILE;
> LOAD DATA LOCAL INPATH "${hiveconf:test.data.dir}/kv1.txt" OVERWRITE INTO TABLE srcpart PARTITION (ds="2008-04-09", hr="11");
> select * from srcpart;
> {code}
> I did not see the issue being hit by non-partitioned/textfile tables, but that is just the luck of the path taken by the code. Inspection of my PG metastore shows all the CLOB fields suffering from this issue.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)