spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: spark data frame write.mode("append") bug
Date Sat, 12 Dec 2015 23:14:33 GMT
If you want to contribute to the project open a JIRA/PR:
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark

On Sat, Dec 12, 2015 at 3:13 AM, Kali.tummala@gmail.com <
Kali.tummala@gmail.com> wrote:

> Hi All,
>
>
> https://github.com/apache/spark/blob/branch-1.5/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L48
>
> In Present spark version in line 48 there is a bug, to check whether table
> exists in a database using limit doesnt work for all databases sql server
> for example.
>
> best way to check whehter table exists in any database is to use, select *
> from table where 1=2;  or select 1 from table where 1=2; this supports all
> the databases.
>
> In spark 1.6 can this change be implemented, this lets
> write.mode("append")
> bug to go away.
>
>
>
> def tableExists(conn: Connection, table: String): Boolean = {
>
>     // Somewhat hacky, but there isn't a good way to identify whether a
> table exists for all
>     // SQL database systems, considering "table" could also include the
> database name.
>     Try(conn.prepareStatement(s"SELECT 1 FROM $table LIMIT
> 1").executeQuery().next()).isSuccess
>   }
>
> Solution:-
> def tableExists(conn: Connection, table: String): Boolean = {
>
>     // Somewhat hacky, but there isn't a good way to identify whether a
> table exists for all
>     // SQL database systems, considering "table" could also include the
> database name.
>     Try(conn.prepareStatement(s"SELECT 1 FROM $table where
> 1=2").executeQuery().next()).isSuccess
>   }
>
>
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-data-frame-write-mode-append-bug-tp25650p25693.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message