Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 6 Dec 2017 00:02:00 +0000 (UTC)
From: "Sean Owen (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.13123033.1512516110000.393892.1512518520199@Atlassian.JIRA>
In-Reply-To: <JIRA.13123033.1512516110000@Atlassian.JIRA>
References: <JIRA.13123033.1512516110000@Atlassian.JIRA> <JIRA.13123033.1512516110643@jira-lw-us.apache.org>
Subject: [jira] [Updated] (SPARK-22706) Cannot read Teradata CLOB column
 type correctly in Spark 2.2.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 06 Dec 2017 00:02:07 -0000


     [ https://issues.apache.org/jira/browse/SPARK-22706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated SPARK-22706:
------------------------------
    Priority: Minor  (was: Major)

On its surface, it looks like it's because the driver's CLOB implementation class doesn't implement toString(), or that its ResultSet.getString doesn't do what you want for a CLOB. That is I think this is a Teradata driver issue, but can't say I'm 100% sure.

> Cannot read Teradata CLOB column type correctly in Spark 2.2.0
> --------------------------------------------------------------
>
>                 Key: SPARK-22706
>                 URL: https://issues.apache.org/jira/browse/SPARK-22706
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0
>         Environment: Spark 2.2.0 
> Teradata driver version: 16.20.00.02 (terajdbc4.jar)
> I am testing with Mac
>            Reporter: Nannan Yu
>            Priority: Minor
>
> When I try to read the table from Teradata dababase with URL like "jdbc:teradata://IP/Database".
> It works well for most of the column datatype.
> But for the CLOB datatype, it cannot read the correct contest: 
>  If I am using Spark:  it will get some result like this:  
>    -----------------------------------------------------------------------
>    DT_INT |  DT_CLOB   
>    ------------------------------------------------------------------------------
>      1         | com.teradata.jdbc.jdk6_SQL_Clob@3cc9ee67
>      2         | com.teradata.jdbc.jdk6_SQL_Clob@7cf9ecad
>    --------------------------------------------------------------------------------
> Actually, it should show the real Character Large Object  as this:
>   -----------------------------------------------------------------------
>    DT_INT |  DT_CLOB   
>    ------------------------------------------------------------------------------
>      1         | JESSIE
>      2         | CATHY
>    --------------------------------------------------------------------------------
> Note: This issue is not for all the 'CLOB' column datatype, for example, DB2-CLOB datatype works fine with Spark. 
> For teradata, we have this issue maybe related to this reason: 
>   https://www.dwhpro.com/teradata-clob/
>    " The data of a CLOB column is stored {color:#f6c342}as a separate sub-table{color}. " for TeraData CLOB? 
> From the Spark SourceCode in the file: org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
>      * case StringType* =>
>       (rs: ResultSet, row: InternalRow, pos: Int) =>
>         // TODO(davies): use getBytes for better performance, if the encoding is UTF-8
>         row.update(pos, UTF8String.fromString(rs.getString(pos + 1)))
> It seems we cannot use this way for the CLOB column from Teradata.
> Actually, I tried to use some other way to read the CLOB data from Teradata, it works well. Also it also works for the DB2-CLOB data. 
>    case StringType =>
>       (rs: ResultSet, row: InternalRow, pos: Int) => {
>         val sqlType = rs.getMetaData.getColumnType(pos + 1)
>         if (java.sql.Types.CLOB == sqlType) {
>           val clobData = rs.getClob(pos + 1);
>           if (clobData != null)
>             row.update(pos, UTF8String.fromString(clobData.getSubString(1, clobData.length().toInt)))
>           else
>             row.update(pos, UTF8String.fromString(rs.getString(pos + 1)))
>         }
>         else {
>           // TODO(davies): use getBytes for better performance, if the encoding is UTF-8
>           row.update(pos, UTF8String.fromString(rs.getString(pos + 1)))
>         }
>       }
> Should we handle the CLOB datatype differently with the normal 'StringType'? 


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org