spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anand Mohan Tumuluri (JIRA)" <>
Subject [jira] [Commented] (SPARK-5472) Add support for reading from and writing to a JDBC database
Date Wed, 04 Feb 2015 00:35:35 GMT


Anand Mohan Tumuluri commented on SPARK-5472:

Thanks again [~rxin] and [~tmyklebu]. My bad, I only checked the table creation scripts in
before and made assumptions.
This would very well satisfy our use case.
The custom partitioning conditions would remove the need to use SQL conditionals as well.

One more question, how do I 'get' new data that got inserted in the source table(s)? Would
'refresh table' work for this?

> Add support for reading from and writing to a JDBC database
> -----------------------------------------------------------
>                 Key: SPARK-5472
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Tor Myklebust
>            Assignee: Tor Myklebust
>            Priority: Blocker
>             Fix For: 1.3.0
> It would be nice to be able to make a table in a JDBC database appear as a table in Spark
SQL.  This would let users, for instance, perform a JOIN between a DataFrame in Spark SQL
with a table in a Postgres database.
> It might also be nice to be able to go the other direction -- save a DataFrame to a database
-- for instance in an ETL job.
> Edited to clarify:  Both of these tasks are certainly possible to accomplish at the moment
with a little bit of ad-hoc glue code.  However, there is no fundamental reason why the user
should need to supply the table schema and some code for pulling data out of a ResultSet row
into a Catalyst Row structure when this information can be derived from the schema of the
database table itself.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message