spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-10756) DataFrame write to teradata using jdbc not working, tries to create table each time irrespective of table existence
Date Tue, 22 Sep 2015 09:48:04 GMT

     [ https://issues.apache.org/jira/browse/SPARK-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Amit updated SPARK-10756:
-------------------------
    Description: 
DataFrame write to teradata using jdbc not working, tries to create table each time irrespective
of table existence. 

Whenever it goes to persist dataframe it checks for the table existence by using query "SELECT
1 FROM $table LIMIT 1" and the keyword limit is not supported in teradata. So the exception
thrown by teradata for keyword is understood as exception for table not exist in Spark and
then Spark runs the create table command irrespective of table was present. 

So Create table command execution fails with the exception of table already exist hence saving
of data frame fails 

Below is the method of JDBCUtils class

/**
   * Returns true if the table already exists in the JDBC database.
   */
  def tableExists(conn: Connection, table: String): Boolean = {
    // Somewhat hacky, but there isn't a good way to identify whether a table exists for all
    // SQL database systems, considering "table" could also include the database name.
    Try(conn.prepareStatement(s"SELECT 1 FROM $table LIMIT 1").executeQuery().next()).isSuccess
  }

In case of teradata, It returns false for every save/write operation irrespective of the fact
that table was present.

  was:
DataFrame write to teradata using jdbc not working, tries to create table each time irrespective
of table existence. 

Whenever it goes to persist dataframe it checks for the table existence by using query "SELECT
1 FROM $table LIMIT 1" and the keyword limit is not supported in teradata. So the exception
thrown by teradata for keyword is understood as exception for table not exist in Spark and
then Spark runs the create table command irrespective of table was present. 

So Create table command execution fails with the exception of table already exist hence saving
of data frame fails 


> DataFrame write to teradata using jdbc not working, tries to create table each time irrespective
of table existence
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-10756
>                 URL: https://issues.apache.org/jira/browse/SPARK-10756
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.5.0
>            Reporter: Amit
>
> DataFrame write to teradata using jdbc not working, tries to create table each time irrespective
of table existence. 
> Whenever it goes to persist dataframe it checks for the table existence by using query
"SELECT 1 FROM $table LIMIT 1" and the keyword limit is not supported in teradata. So the
exception thrown by teradata for keyword is understood as exception for table not exist in
Spark and then Spark runs the create table command irrespective of table was present. 
> So Create table command execution fails with the exception of table already exist hence
saving of data frame fails 
> Below is the method of JDBCUtils class
> /**
>    * Returns true if the table already exists in the JDBC database.
>    */
>   def tableExists(conn: Connection, table: String): Boolean = {
>     // Somewhat hacky, but there isn't a good way to identify whether a table exists
for all
>     // SQL database systems, considering "table" could also include the database name.
>     Try(conn.prepareStatement(s"SELECT 1 FROM $table LIMIT 1").executeQuery().next()).isSuccess
>   }
> In case of teradata, It returns false for every save/write operation irrespective of
the fact that table was present.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message